Sep 11 16:17:04.209124 ip-10-0-137-32 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:17:04.571326 ip-10-0-137-32 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:04.571326 ip-10-0-137-32 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:17:04.571326 ip-10-0-137-32 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:04.571326 ip-10-0-137-32 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:17:04.571326 ip-10-0-137-32 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:04.572906 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.572811    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:17:04.576904 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576881    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:04.576904 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576900    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:04.576904 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576904    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:04.576904 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576907    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:04.576904 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576910    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:04.576904 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576913    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576916    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576919    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576922    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576925    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576927    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576931    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576934    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576936    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576939    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576942    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576944    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576947    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576950    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576952    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576955    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576958    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576960    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576963    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576965    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:04.577118 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576968    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576970    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576973    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576975    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576978    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576981    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576988    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576992    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576995    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.576998    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577002    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577005    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577008    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577010    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577014    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577017    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577020    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577024    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577028    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:04.577598 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577031    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577034    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577037    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577041    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577044    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577046    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577050    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577052    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577055    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577057    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577061    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577065    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577068    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577072    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577074    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577077    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577079    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577082    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577085    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577087    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:04.578135 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577090    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577092    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577095    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577097    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577100    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577102    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577105    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577109    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577112    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577115    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577118    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577120    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577123    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577126    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577129    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577132    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577134    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577137    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577139    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577142    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:04.578632 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577145    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577148    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577150    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577152    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577155    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577157    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577160    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577162    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577165    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577168    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577170    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577173    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577175    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577178    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577180    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577889    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577897    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577901    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577904    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577907    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:04.579201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577910    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577914    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577918    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577920    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577924    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577926    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577930    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577933    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577935    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577938    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577941    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577944    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577947    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577950    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577953    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577955    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577958    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577961    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577964    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577966    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:04.579695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577969    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577971    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577974    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577976    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577979    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577982    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577985    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577987    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577990    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577993    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577996    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.577999    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578002    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578005    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578007    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578010    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578012    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578015    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578018    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578021    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:04.580218 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578023    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578026    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578028    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578031    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578033    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578036    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578039    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578042    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578046    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578050    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578053    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578056    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578059    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578062    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578065    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578067    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578070    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578072    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578075    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:04.580705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578079    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578084    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578087    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578089    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578092    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578095    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578097    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578100    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578102    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578105    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578107    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578110    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578112    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578115    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578118    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578120    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578123    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578125    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578127    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578131    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:04.581192 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578135    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578138    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578141    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578144    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578147    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578149    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578152    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578154    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578157    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578159    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578162    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578164    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578167    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578171    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578175    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578531    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578537    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578540    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578543    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578546    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:04.581689 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578549    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578552    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578556    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578559    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578561    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578564    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578567    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578570    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578573    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578577    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578580    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578583    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578586    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578588    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578591    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578593    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578595    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578601    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578603    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578606    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:04.582194 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578608    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578611    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578615    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578618    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578621    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578625    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578627    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578630    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578634    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578637    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578640    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578643    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578645    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578648    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578650    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578653    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578656    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578659    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578661    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578664    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:04.582688 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578666    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578669    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578671    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578674    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578677    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578679    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578682    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578684    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578704    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578708    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578711    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578714    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578717    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578720    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578723    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578726    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578728    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578731    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578734    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578737    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:04.583173 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578739    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578742    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578744    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578747    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578749    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578752    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578754    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578757    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578760    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578766    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578769    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578771    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578774    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578776    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578779    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578783    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578786    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578789    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578792    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:04.583657 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578795    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578798    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578800    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578803    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578806    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578808    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578811    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578813    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578816    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578819    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578836    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578839    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578841    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578844    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.578846    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578914    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578954    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578960    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578965    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578969    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:17:04.584199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578972    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578976    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578981    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578984    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578987    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578990    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578994    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.578997    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579000    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579003    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579006    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579010    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579013    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579018    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579021    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579024    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579027    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579030    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579034    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579037    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579040    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579043    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579046    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579049    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:17:04.584689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579052    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579056    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579059    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579063    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579066    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579069    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579072    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579075    2580 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579080    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579085    2580 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579088    2580 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579091    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579094    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579097    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579101    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579104    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579107    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579110    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579113    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579116    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579119    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579122    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579125    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579128    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579131    2580 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:17:04.585277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579135    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579138    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579141    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579143    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579146    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579149    2580 flags.go:64] FLAG: --help="false"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579152    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-137-32.ec2.internal"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579155    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579158    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579161    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579164    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579167    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579170    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579173    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579176    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579180    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579183    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579186    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579189    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579192    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579195    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579198    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579200    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579204    2580 flags.go:64] FLAG: --lock-file=""
Sep 11 16:17:04.585998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579207    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579210    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579213    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579218    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579221    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579226    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579229    2580 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579233    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579236    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579239    2580 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579242    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579246    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579249    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579253    2580 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579256    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579259    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579262    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579264    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579267    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579271    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579274    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579281    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579284    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579289    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:17:04.586580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579292    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579295    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579300    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579303    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579306    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579309    2580 flags.go:64] FLAG: --port="10250"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579312    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579315    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d07fdcfe64bbdfae"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579318    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579321    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579324    2580 flags.go:64] FLAG: --register-node="true"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579329    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579337    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579343    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579349    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579353    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579359    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579363    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579367    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579370    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579372    2580 flags.go:64] FLAG: --runonce="false"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579375    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579379    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579382    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579385    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:17:04.587193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579388    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579390    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579393    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579396    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579399    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579402    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579407    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579410    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579414    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579417    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579420    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579425    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579428    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579431    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579434    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579437    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579440    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579443    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579446    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579449    2580 flags.go:64] FLAG: --v="2"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579453    2580 flags.go:64] FLAG: --version="false"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579458    2580 flags.go:64] FLAG: --vmodule=""
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579462    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.579465    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:17:04.587847 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579576    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579581    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579585    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579588    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579592    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579594    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579597    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579600    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579603    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579605    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579608    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579610    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579613    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579616    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579619    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579622    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579625    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579627    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579630    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:04.588430 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579633    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579635    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579638    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579640    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579643    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579646    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579649    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579652    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579655    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579658    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579664    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579667    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579670    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579672    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579675    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579678    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579680    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579683    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579686    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579688    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:04.588914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579691    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579693    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579696    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579699    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579701    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579704    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579708    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579716    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579719    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579722    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579725    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579727    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579730    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579732    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579735    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579737    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579740    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579742    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579745    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579747    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:04.589396 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579750    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579752    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579756    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579759    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579762    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579765    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579767    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579770    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579772    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579775    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579778    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579780    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579783    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579786    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579788    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579792    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579795    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579798    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579800    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:04.589913 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579805    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579807    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579810    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579813    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579815    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579818    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579832    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579835    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579838    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579840    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579843    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579846    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579850    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579853    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579856    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579860    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579863    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579866    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579869    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:04.590382 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579872    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.579874    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.580414    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.587095    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.587111    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.588441    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.588449    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588510    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588517    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588521    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588524    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588528    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588531    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588534    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:04.590866 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588537    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588539    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588542    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588545    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588548    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588550    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588553    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588556    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588558    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588561    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588564    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588566    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588569    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588571    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588574    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588577    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588581    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588584    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588588    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:04.591223 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588590    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588593    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588596    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588600    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588603    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588606    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588608    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588611    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588614    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588616    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588618    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588621    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588624    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588627    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588629    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588632    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588634    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588637    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588640    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588643    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588645    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:04.591682 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588648    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588651    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588653    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588656    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588658    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588661    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588664    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588666    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588669    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588671    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588675    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588678    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588680    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588683    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588687    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588691    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588694    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588696    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588699    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588701    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:04.592201 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588704    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588706    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588709    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588712    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588714    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588717    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588720    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588722    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588725    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588727    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588730    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588732    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588735    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588738    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588740    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588743    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588745    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588748    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588751    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588755    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:04.592679 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588759    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588761    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588764    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588767    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588770    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588773    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588776    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588779    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588782    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588784    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588787    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588790    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.588795    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588922    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:04.593187 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588928    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588932    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588936    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588938    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588942    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588945    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588948    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588950    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588953    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588956    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588959    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588962    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588965    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588967    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588970    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588972    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588975    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588977    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588980    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588983    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:04.593533 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588985    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588989    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588992    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588994    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.588997    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589000    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589003    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589006    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589008    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589012    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589016    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589018    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589021    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589024    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589026    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589029    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589032    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589034    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589037    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:04.594046 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589039    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589042    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589045    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589047    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589050    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589052    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589055    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589060    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589063    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589065    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589068    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589070    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589073    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589076    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589078    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589081    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589084    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589087    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589089    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:04.594498 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589092    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589095    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589098    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589100    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589103    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589105    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589108    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589111    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589113    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589116    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589118    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589121    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589123    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589126    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589128    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589131    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589133    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589137    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589140    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:04.595005 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589142    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589145    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589147    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589150    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589152    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589155    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589158    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589160    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589163    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589166    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589168    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589171    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589173    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589176    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589179    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589182    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589185    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589187    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589190    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:04.595468 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:04.589193    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.589198    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.589309    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.591218    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.592139    2580 server.go:1019] "Starting client certificate rotation"
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.592237    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.592276    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:17:04.595930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.594536    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:17:04.596980 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.596956    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:17:04.609256 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.609237    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:17:04.612482 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.612464    2580 log.go:25] "Validated CRI v1 image API"
Sep 11 16:17:04.615209 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.615193    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:17:04.620438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.620418    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e2e040ad-c34f-4ccd-bd71-33fab58630f6:/dev/nvme0n1p4 f84f6381-38a4-41ef-b28d-5e98836a0f6d:/dev/nvme0n1p3]
Sep 11 16:17:04.620484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.620438    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:17:04.625809 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.625788    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:17:04.626772 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.626672    2580 manager.go:216] Machine: {Timestamp:2026-09-11 16:17:04.625312514 +0000 UTC m=+0.306416931 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3057775 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2c534652e2b03be15892ade6e16f SystemUUID:ec2a2c53-4652-e2b0-3be1-5892ade6e16f BootID:093f93bf-6322-470e-855f-09771dc35419 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fe:51:33:09:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fe:51:33:09:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:b9:90:23:13:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:17:04.626772 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.626771    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:17:04.626909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.626866    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:17:04.629561 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.629530    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:17:04.629699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.629565    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:17:04.630830 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.630811    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:17:04.630863 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.630835    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:17:04.630863 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.630842    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:17:04.631498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.631488    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:17:04.632230 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.632220    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:17:04.635712 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.635697    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:17:04.635835 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.635813    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:17:04.636335 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636324    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:17:04.636376 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636341    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:17:04.636376 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636359    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:17:04.636376 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636368    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:17:04.636376 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636376    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:17:04.636737 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636725    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:17:04.636779 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.636744    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:17:04.638838 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.638806    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:17:04.641305 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.641291    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:17:04.641367 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.641321    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:17:04.643883 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643869    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643889    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643896    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643901    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643910    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643916    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643930    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643936    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:17:04.643943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643942    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:17:04.644155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643948    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:17:04.644155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643956    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:17:04.644155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.643961    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:17:04.644919 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.644909    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:17:04.644919 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.644919    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:17:04.648438 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.648409    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:04.648517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.648480    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:04.648676 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.648654    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:04.648798 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.648787    2580 server.go:1268] "Started kubelet"
Sep 11 16:17:04.648967 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.648895    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:17:04.649020 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.648895    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:17:04.649020 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.649008    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:17:04.653499 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.653466    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:17:04.653735 ip-10-0-137-32 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:17:04.657226 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.657150    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:17:04.657226 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.657159    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:17:04.659673 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.658625    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-32.ec2.internal.18d45077266a4d34  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-32.ec2.internal,UID:ip-10-0-137-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-32.ec2.internal,},FirstTimestamp:2026-09-11 16:17:04.648760628 +0000 UTC m=+0.329865044,LastTimestamp:2026-09-11 16:17:04.648760628 +0000 UTC m=+0.329865044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-32.ec2.internal,}"
Sep 11 16:17:04.660798 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.660783    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vdk5n"
Sep 11 16:17:04.663046 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663029    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:17:04.663046 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663038    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:17:04.663193 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.663123    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:04.663744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663729    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:17:04.663848 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663731    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:17:04.663848 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663757    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:17:04.663848 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663795    2580 factory.go:55] Registering systemd factory
Sep 11 16:17:04.663848 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663810    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:17:04.664018 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663883    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:17:04.664018 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.663893    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:17:04.664102 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.664090    2580 factory.go:153] Registering CRI-O factory
Sep 11 16:17:04.664141 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.664103    2580 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:17:04.664191 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.664142    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:17:04.664191 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.664156    2580 factory.go:103] Registering Raw factory
Sep 11 16:17:04.664191 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.664166    2580 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:17:04.664542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.664530    2580 manager.go:349] Starting recovery of all containers
Sep 11 16:17:04.668552 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.668527    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vdk5n"
Sep 11 16:17:04.668740 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.668720    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:04.668907 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.668888    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:17:04.671147 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.671123    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:17:04.673965 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.673948    2580 manager.go:354] Recovery completed
Sep 11 16:17:04.680004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.679842    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:04.682723 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.682692    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:04.682793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.682734    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:04.682793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.682748    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:04.683348 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.683335    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:17:04.683348 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.683347    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:17:04.683431 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.683361    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:17:04.683468 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.683340    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:04.684125 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.684105    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:04.684206 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.684140    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:04.684206 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.684154    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:04.684724 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.684713    2580 policy_none.go:49] "None policy: Start"
Sep 11 16:17:04.684767 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.684729    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:17:04.684767 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.684738    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:17:04.685521 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.685208    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-32.ec2.internal.18d4507728706189  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-32.ec2.internal,UID:ip-10-0-137-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-32.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-32.ec2.internal,},FirstTimestamp:2026-09-11 16:17:04.682713481 +0000 UTC m=+0.363817904,LastTimestamp:2026-09-11 16:17:04.682713481 +0000 UTC m=+0.363817904,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-32.ec2.internal,}"
Sep 11 16:17:04.687233 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.687216    2580 policy_none.go:47] "Start"
Sep 11 16:17:04.721373 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.721355    2580 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:17:04.721486 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.721409    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:17:04.721486 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.721422    2580 server.go:87] "Starting device plugin registration server"
Sep 11 16:17:04.721775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.721761    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:17:04.721855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.721775    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:17:04.721945 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.721929    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:17:04.722035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.722006    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:17:04.722035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.722019    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:17:04.722593 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.722571    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:17:04.722674 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.722627    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:04.740281 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.740259    2580 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2614 into the system container \"/system.slice\": <nil>"
Sep 11 16:17:04.822528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.822434    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:04.823449 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.823432    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:04.823554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.823461    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:04.823554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.823470    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:04.823554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.823494    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-32.ec2.internal"
Sep 11 16:17:04.831520 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.831503    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-32.ec2.internal"
Sep 11 16:17:04.831564 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.831528    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-32.ec2.internal\": node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:04.848960 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.848941    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:04.934031 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.934000    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:17:04.934190 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.934062    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:17:04.934190 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.934091    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:17:04.934300 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.934196    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:17:04.939809 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:04.939785    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:04.949768 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:04.949752    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.035187 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.035153    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"]
Sep 11 16:17:05.035327 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.035251    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.036808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.036793    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.036902 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.036843    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.036902 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.036855    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.038045 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038033    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.038151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.038207 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038166    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.038750 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038735    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.038750 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038754    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.038911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038764    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.038911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038838    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.038911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038868    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.038911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.038884    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.040004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.039989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.040060 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.040015    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.040681 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.040664    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.040761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.040694    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.040761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.040706    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.050175 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.050161    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.065443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.065418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20e5ec496ea128525a2252a4ae0e8d2f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-32.ec2.internal\" (UID: \"20e5ec496ea128525a2252a4ae0e8d2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.065548 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.065444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5c5a235530abfb8b7c3ed471e06b88-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal\" (UID: \"4f5c5a235530abfb8b7c3ed471e06b88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.065548 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.065464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5c5a235530abfb8b7c3ed471e06b88-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal\" (UID: \"4f5c5a235530abfb8b7c3ed471e06b88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.065548 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.065478    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-32.ec2.internal\" not found" node="ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.070028 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.070014    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-32.ec2.internal\" not found" node="ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.151011 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.150929    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.166262 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.166242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20e5ec496ea128525a2252a4ae0e8d2f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-32.ec2.internal\" (UID: \"20e5ec496ea128525a2252a4ae0e8d2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.166352 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.166267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5c5a235530abfb8b7c3ed471e06b88-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal\" (UID: \"4f5c5a235530abfb8b7c3ed471e06b88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.166352 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.166287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5c5a235530abfb8b7c3ed471e06b88-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal\" (UID: \"4f5c5a235530abfb8b7c3ed471e06b88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.166438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.166365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20e5ec496ea128525a2252a4ae0e8d2f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-32.ec2.internal\" (UID: \"20e5ec496ea128525a2252a4ae0e8d2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.166438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.166367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5c5a235530abfb8b7c3ed471e06b88-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal\" (UID: \"4f5c5a235530abfb8b7c3ed471e06b88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.166438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.166366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5c5a235530abfb8b7c3ed471e06b88-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal\" (UID: \"4f5c5a235530abfb8b7c3ed471e06b88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.251454 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.251413    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.352340 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.352306    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.367970 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.367939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.372967 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.372944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.455739 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.455647    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.502551 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.502531    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:05.559530 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.559493    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.592374 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.592344    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:17:05.592846 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.592546    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:17:05.592846 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.592546    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:17:05.660533 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:05.660501    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-32.ec2.internal\" not found"
Sep 11 16:17:05.662950 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.662929    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:05.663467 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.663448    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.663520 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.663483    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:17:05.673664 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.673646    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:17:05.674772 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.674747    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:12:04 +0000 UTC" deadline="2028-02-18 00:19:50.350555808 +0000 UTC"
Sep 11 16:17:05.674823 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.674774    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12584h2m44.675785736s"
Sep 11 16:17:05.675385 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.675367    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal"
Sep 11 16:17:05.677784 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.677769    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:17:05.689262 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.689242    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:17:05.697999 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.697981    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxfzj"
Sep 11 16:17:05.704711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.704694    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxfzj"
Sep 11 16:17:05.813407 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.813096    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:05.932120 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:05.932090    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5c5a235530abfb8b7c3ed471e06b88.slice/crio-68096ede8d3dab9e7d6d458aa9eb8fd35a674eb55927ddbe9dd6ba39e65ffd37 WatchSource:0}: Error finding container 68096ede8d3dab9e7d6d458aa9eb8fd35a674eb55927ddbe9dd6ba39e65ffd37: Status 404 returned error can't find the container with id 68096ede8d3dab9e7d6d458aa9eb8fd35a674eb55927ddbe9dd6ba39e65ffd37
Sep 11 16:17:05.932576 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:05.932552    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e5ec496ea128525a2252a4ae0e8d2f.slice/crio-1b8e5d31740ba68e005e8a47750b60bc93e83eee7cadf9f57f41dabf6128a537 WatchSource:0}: Error finding container 1b8e5d31740ba68e005e8a47750b60bc93e83eee7cadf9f57f41dabf6128a537: Status 404 returned error can't find the container with id 1b8e5d31740ba68e005e8a47750b60bc93e83eee7cadf9f57f41dabf6128a537
Sep 11 16:17:05.937860 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.937842    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:17:05.938181 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.938121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal" event={"ID":"4f5c5a235530abfb8b7c3ed471e06b88","Type":"ContainerStarted","Data":"68096ede8d3dab9e7d6d458aa9eb8fd35a674eb55927ddbe9dd6ba39e65ffd37"}
Sep 11 16:17:05.939448 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:05.939427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal" event={"ID":"20e5ec496ea128525a2252a4ae0e8d2f","Type":"ContainerStarted","Data":"1b8e5d31740ba68e005e8a47750b60bc93e83eee7cadf9f57f41dabf6128a537"}
Sep 11 16:17:06.556564 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.556528    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:06.640524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.640222    2580 apiserver.go:52] "Watching apiserver"
Sep 11 16:17:06.651241 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.651211    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:17:06.652137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.651743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-bdf6j","openshift-network-operator/iptables-alerter-cnqfb","kube-system/konnectivity-agent-2wl27","kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal","openshift-image-registry/node-ca-qm7s8","openshift-multus/multus-5b6cb","openshift-multus/network-metrics-daemon-lhpp4","openshift-ovn-kubernetes/ovnkube-node-s9f5g","kube-system/global-pull-secret-syncer-gmprr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml","openshift-cluster-node-tuning-operator/tuned-cnvx5","openshift-dns/node-resolver-v5fvg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal","openshift-multus/multus-additional-cni-plugins-jhltr"]
Sep 11 16:17:06.653850 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.653385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:06.653850 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.653481    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:06.654873 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.654820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.656300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.655983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.659290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.660798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.660818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sm5bl\""
Sep 11 16:17:06.661689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xb992\""
Sep 11 16:17:06.662131 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:06.662131 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661764    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:17:06.662131 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.661771    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:06.662131 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.661956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.662446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.662426    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:17:06.662753 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.662732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vnj9l\""
Sep 11 16:17:06.662970 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.662954    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.663224 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.663198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.663588 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.663571    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.665019 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.664839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.665019 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.664908    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:06.665736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.665318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lbjjm\""
Sep 11 16:17:06.665736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.665348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:17:06.665736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.665422    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.665736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.665350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.665736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.665548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:17:06.665736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.665718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.667300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.666735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:17:06.667300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.666822    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:17:06.667300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.666831    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.667300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.666992    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ksmmm\""
Sep 11 16:17:06.667300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.667006    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:17:06.667300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.667127    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.667638 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.667543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.671480 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.671455    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.672386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.672871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.672984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.673203    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tbbhq\""
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.673573    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.673976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shf5t\""
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da9490e-eee0-48dc-b34e-d00efe02064f-host-slash\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/3da9490e-eee0-48dc-b34e-d00efe02064f-kube-api-access-b4rlq\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-ovnkube-script-lib\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-cni-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-socket-dir-parent\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.674749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-daemon-config\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-slash\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674774    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-systemd\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-ovnkube-config\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.674970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7th\" (UniqueName: \"kubernetes.io/projected/d91b4ccf-9425-4777-9bac-32be72517058-kube-api-access-hk7th\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-multus-certs\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-kubelet-config\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-etc-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-env-overrides\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-hostroot\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-etc-kubernetes\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.675358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2tj\" (UniqueName: \"kubernetes.io/projected/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-kube-api-access-hg2tj\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-kubelet\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-registration-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b264b7cd-f54d-4f10-b644-22fc59c4ab43-konnectivity-ca\") pod \"konnectivity-agent-2wl27\" (UID: \"b264b7cd-f54d-4f10-b644-22fc59c4ab43\") " pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tp97\" (UniqueName: \"kubernetes.io/projected/8cd81b90-21f6-4343-99c5-3d746e3c480e-kube-api-access-7tp97\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-cni-multus\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.675929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-kubelet\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.676275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.675892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-device-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.676275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-cni-netd\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676032    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:17:06.676275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d91b4ccf-9425-4777-9bac-32be72517058-ovn-node-metrics-cert\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d65744c-640a-47cf-bd5d-966db47417af-host\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.676528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-socket-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.676528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-run-netns\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676682 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-log-socket\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676682 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.676682 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-var-lib-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676832 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.676832 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676980 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.676980 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-node-log\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.676980 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-dbus\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.676980 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-systemd-units\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.677174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.676954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-k8s-cni-cncf-io\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-sys-fs\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.677174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da9490e-eee0-48dc-b34e-d00efe02064f-iptables-alerter-script\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.677174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-system-cni-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-cni-binary-copy\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-conf-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677222    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-cni-bin\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b264b7cd-f54d-4f10-b644-22fc59c4ab43-agent-certs\") pod \"konnectivity-agent-2wl27\" (UID: \"b264b7cd-f54d-4f10-b644-22fc59c4ab43\") " pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-ovn\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpnx\" (UniqueName: \"kubernetes.io/projected/7d65744c-640a-47cf-bd5d-966db47417af-kube-api-access-mlpnx\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-netns\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d65744c-640a-47cf-bd5d-966db47417af-serviceca\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.677470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-cnibin\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:06.677998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6fgnx\""
Sep 11 16:17:06.677998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-cni-bin\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.677998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-etc-selinux\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.677998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.677579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6qt\" (UniqueName: \"kubernetes.io/projected/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-kube-api-access-mk6qt\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.680673 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.680462    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:17:06.680673 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.680488    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kc75s\""
Sep 11 16:17:06.680673 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.680491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:17:06.706815 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.706755    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:12:05 +0000 UTC" deadline="2028-02-13 06:51:30.286901423 +0000 UTC"
Sep 11 16:17:06.706815 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.706816    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12470h34m23.580089619s"
Sep 11 16:17:06.765083 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.765052    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:17:06.778598 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-cni-bin\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b264b7cd-f54d-4f10-b644-22fc59c4ab43-agent-certs\") pod \"konnectivity-agent-2wl27\" (UID: \"b264b7cd-f54d-4f10-b644-22fc59c4ab43\") " pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-ovn\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpnx\" (UniqueName: \"kubernetes.io/projected/7d65744c-640a-47cf-bd5d-966db47417af-kube-api-access-mlpnx\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-netns\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-tuned\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.778744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d65744c-640a-47cf-bd5d-966db47417af-serviceca\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-cnibin\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-system-cni-dir\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysctl-d\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-var-lib-kubelet\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-cni-bin\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-etc-selinux\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6qt\" (UniqueName: \"kubernetes.io/projected/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-kube-api-access-mk6qt\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-kubernetes\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-host\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.778992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da9490e-eee0-48dc-b34e-d00efe02064f-host-slash\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/3da9490e-eee0-48dc-b34e-d00efe02064f-kube-api-access-b4rlq\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-ovnkube-script-lib\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-cni-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-socket-dir-parent\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-daemon-config\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.779152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kxh\" (UniqueName: \"kubernetes.io/projected/1e9c8529-29e0-4e00-b061-becbd28e2f1b-kube-api-access-d2kxh\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-slash\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-systemd\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-ovnkube-config\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7th\" (UniqueName: \"kubernetes.io/projected/d91b4ccf-9425-4777-9bac-32be72517058-kube-api-access-hk7th\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-multus-certs\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-kubelet-config\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-etc-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-env-overrides\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-hostroot\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-etc-kubernetes\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2tj\" (UniqueName: \"kubernetes.io/projected/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-kube-api-access-hg2tj\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8529-29e0-4e00-b061-becbd28e2f1b-tmp-dir\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-kubelet\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-registration-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-lib-modules\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.780049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b264b7cd-f54d-4f10-b644-22fc59c4ab43-konnectivity-ca\") pod \"konnectivity-agent-2wl27\" (UID: \"b264b7cd-f54d-4f10-b644-22fc59c4ab43\") " pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tp97\" (UniqueName: \"kubernetes.io/projected/8cd81b90-21f6-4343-99c5-3d746e3c480e-kube-api-access-7tp97\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-cni-multus\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-kubelet\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-device-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-cni-netd\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d91b4ccf-9425-4777-9bac-32be72517058-ovn-node-metrics-cert\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d65744c-640a-47cf-bd5d-966db47417af-host\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-socket-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-cni-binary-copy\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-run-netns\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-log-socket\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysconfig\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.780748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-var-lib-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhlj\" (UniqueName: \"kubernetes.io/projected/58c23413-8a02-469b-bf68-39ddd5729968-kube-api-access-tqhlj\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-modprobe-d\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-node-log\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-systemd\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktz9n\" (UniqueName: \"kubernetes.io/projected/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-kube-api-access-ktz9n\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-dbus\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.779972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysctl-conf\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-systemd-units\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-k8s-cni-cncf-io\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-sys-fs\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-run\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-tmp\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da9490e-eee0-48dc-b34e-d00efe02064f-iptables-alerter-script\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-system-cni-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.781364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-cni-binary-copy\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-conf-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9c8529-29e0-4e00-b061-becbd28e2f1b-hosts-file\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-cnibin\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-sys\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-run-ovn-kubernetes\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.780461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-cni-bin\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.780748    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.780822    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.280797532 +0000 UTC m=+2.961901942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.781015    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.781073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-ovn\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.781358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-netns\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.781439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-slash\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.781487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-systemd\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.782205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.781871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-daemon-config\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.782276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d65744c-640a-47cf-bd5d-966db47417af-serviceca\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.783037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.782353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da9490e-eee0-48dc-b34e-d00efe02064f-host-slash\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.783037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.782425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-cni-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.782428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-cnibin\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.782471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-socket-dir-parent\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.782994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-multus-certs\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-ovnkube-script-lib\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.783363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-kubelet-config\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.783363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-etc-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.783363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-etc-selinux\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.783363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-cni-bin\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-hostroot\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-etc-kubernetes\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.783611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-env-overrides\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.783765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-kubelet\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.783765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-registration-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.783869 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-sys-fs\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.783950 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b264b7cd-f54d-4f10-b644-22fc59c4ab43-konnectivity-ca\") pod \"konnectivity-agent-2wl27\" (UID: \"b264b7cd-f54d-4f10-b644-22fc59c4ab43\") " pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.784003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-node-log\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.783966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d91b4ccf-9425-4777-9bac-32be72517058-ovnkube-config\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784164 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-systemd-units\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784164 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-run-k8s-cni-cncf-io\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.784164 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-dbus\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:06.784318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-kubelet\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.784318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-device-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.784428 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-cni-netd\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784479 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-run-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-log-socket\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784639 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.784620    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:06.784694 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da9490e-eee0-48dc-b34e-d00efe02064f-iptables-alerter-script\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.784694 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-cni-binary-copy\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.784694 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.784690    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.284672417 +0000 UTC m=+2.965776844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:06.784847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-host-var-lib-cni-multus\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.784847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-var-lib-openvswitch\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.784847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-system-cni-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.784847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d65744c-640a-47cf-bd5d-966db47417af-host\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.784847 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d91b4ccf-9425-4777-9bac-32be72517058-host-run-netns\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.785137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-socket-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.785137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.785137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.784963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-multus-conf-dir\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.786888 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.786842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b264b7cd-f54d-4f10-b644-22fc59c4ab43-agent-certs\") pod \"konnectivity-agent-2wl27\" (UID: \"b264b7cd-f54d-4f10-b644-22fc59c4ab43\") " pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:06.787733 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.787650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d91b4ccf-9425-4777-9bac-32be72517058-ovn-node-metrics-cert\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.791149 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.791128    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:06.791254 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.791154    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:06.791254 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.791167    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:06.791254 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:06.791230    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.291202547 +0000 UTC m=+2.972306967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:06.792921 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.792848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6qt\" (UniqueName: \"kubernetes.io/projected/63dce7a7-e9d6-4880-8c61-74ddcd4dea50-kube-api-access-mk6qt\") pod \"aws-ebs-csi-driver-node-glnml\" (UID: \"63dce7a7-e9d6-4880-8c61-74ddcd4dea50\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:06.793859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.793825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/3da9490e-eee0-48dc-b34e-d00efe02064f-kube-api-access-b4rlq\") pod \"iptables-alerter-cnqfb\" (UID: \"3da9490e-eee0-48dc-b34e-d00efe02064f\") " pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.795074 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.795050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tp97\" (UniqueName: \"kubernetes.io/projected/8cd81b90-21f6-4343-99c5-3d746e3c480e-kube-api-access-7tp97\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:06.795269 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.795244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpnx\" (UniqueName: \"kubernetes.io/projected/7d65744c-640a-47cf-bd5d-966db47417af-kube-api-access-mlpnx\") pod \"node-ca-qm7s8\" (UID: \"7d65744c-640a-47cf-bd5d-966db47417af\") " pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:06.796440 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.796400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2tj\" (UniqueName: \"kubernetes.io/projected/c764ae30-1eb4-43a7-a78e-bd975b3fa3e3-kube-api-access-hg2tj\") pod \"multus-5b6cb\" (UID: \"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3\") " pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:06.798247 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.798206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7th\" (UniqueName: \"kubernetes.io/projected/d91b4ccf-9425-4777-9bac-32be72517058-kube-api-access-hk7th\") pod \"ovnkube-node-s9f5g\" (UID: \"d91b4ccf-9425-4777-9bac-32be72517058\") " pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:06.881173 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8529-29e0-4e00-b061-becbd28e2f1b-tmp-dir\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.881173 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-lib-modules\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881405 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-cni-binary-copy\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.881405 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.881499 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-lib-modules\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881541 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9c8529-29e0-4e00-b061-becbd28e2f1b-tmp-dir\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.881541 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.881607 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysconfig\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881607 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhlj\" (UniqueName: \"kubernetes.io/projected/58c23413-8a02-469b-bf68-39ddd5729968-kube-api-access-tqhlj\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.881716 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-modprobe-d\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881716 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-systemd\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881716 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysconfig\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881716 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktz9n\" (UniqueName: \"kubernetes.io/projected/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-kube-api-access-ktz9n\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881716 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysctl-conf\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-run\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-tmp\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.881910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9c8529-29e0-4e00-b061-becbd28e2f1b-hosts-file\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.881910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-cnibin\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.881910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-sys\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-cni-binary-copy\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-systemd\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.881969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-modprobe-d\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9c8529-29e0-4e00-b061-becbd28e2f1b-hosts-file\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-tuned\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysctl-conf\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-run\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-system-cni-dir\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-cnibin\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysctl-d\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-sys\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-var-lib-kubelet\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-system-cni-dir\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-kubernetes\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-host\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-sysctl-d\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kxh\" (UniqueName: \"kubernetes.io/projected/1e9c8529-29e0-4e00-b061-becbd28e2f1b-kube-api-access-d2kxh\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-kubernetes\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-host\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c23413-8a02-469b-bf68-39ddd5729968-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-var-lib-kubelet\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.882833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.882533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c23413-8a02-469b-bf68-39ddd5729968-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.885152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.885126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-etc-tuned\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.886646 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.886622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-tmp\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.892535 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.892512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktz9n\" (UniqueName: \"kubernetes.io/projected/a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4-kube-api-access-ktz9n\") pod \"tuned-cnvx5\" (UID: \"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:06.892911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.892878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhlj\" (UniqueName: \"kubernetes.io/projected/58c23413-8a02-469b-bf68-39ddd5729968-kube-api-access-tqhlj\") pod \"multus-additional-cni-plugins-jhltr\" (UID: \"58c23413-8a02-469b-bf68-39ddd5729968\") " pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:06.893674 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.893645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kxh\" (UniqueName: \"kubernetes.io/projected/1e9c8529-29e0-4e00-b061-becbd28e2f1b-kube-api-access-d2kxh\") pod \"node-resolver-v5fvg\" (UID: \"1e9c8529-29e0-4e00-b061-becbd28e2f1b\") " pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:06.979446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.979413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnqfb"
Sep 11 16:17:06.989914 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:06.989884    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da9490e_eee0_48dc_b34e_d00efe02064f.slice/crio-8ac892e6101efaed75e96ca19a58419a43944ae5ca108d97b8a514a4f8a3f4c8 WatchSource:0}: Error finding container 8ac892e6101efaed75e96ca19a58419a43944ae5ca108d97b8a514a4f8a3f4c8: Status 404 returned error can't find the container with id 8ac892e6101efaed75e96ca19a58419a43944ae5ca108d97b8a514a4f8a3f4c8
Sep 11 16:17:06.992358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:06.992052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:07.001009 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.000980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qm7s8"
Sep 11 16:17:07.001441 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:07.001416    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb264b7cd_f54d_4f10_b644_22fc59c4ab43.slice/crio-c297a7f140a05d5249fcf0da7e4ecbfb61af74a484ce3a210779fd5cceab5cc1 WatchSource:0}: Error finding container c297a7f140a05d5249fcf0da7e4ecbfb61af74a484ce3a210779fd5cceab5cc1: Status 404 returned error can't find the container with id c297a7f140a05d5249fcf0da7e4ecbfb61af74a484ce3a210779fd5cceab5cc1
Sep 11 16:17:07.007404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.007379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5b6cb"
Sep 11 16:17:07.012044 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:07.012008    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d65744c_640a_47cf_bd5d_966db47417af.slice/crio-4c36a09cd121f3a2f65053a5b68da1e194c0a607bbdb525077e794116e2a7e23 WatchSource:0}: Error finding container 4c36a09cd121f3a2f65053a5b68da1e194c0a607bbdb525077e794116e2a7e23: Status 404 returned error can't find the container with id 4c36a09cd121f3a2f65053a5b68da1e194c0a607bbdb525077e794116e2a7e23
Sep 11 16:17:07.016746 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.016592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:07.018251 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:07.018227    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc764ae30_1eb4_43a7_a78e_bd975b3fa3e3.slice/crio-f21bc3aac38e207fc907c57861eb2eb3fe7fc74c4741dcfba5e3933ac341e43f WatchSource:0}: Error finding container f21bc3aac38e207fc907c57861eb2eb3fe7fc74c4741dcfba5e3933ac341e43f: Status 404 returned error can't find the container with id f21bc3aac38e207fc907c57861eb2eb3fe7fc74c4741dcfba5e3933ac341e43f
Sep 11 16:17:07.026628 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:07.026601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91b4ccf_9425_4777_9bac_32be72517058.slice/crio-99563b0bfdcd8ae53a02ea22728e14041933c6e6def0c35f826a751e6314a8e6 WatchSource:0}: Error finding container 99563b0bfdcd8ae53a02ea22728e14041933c6e6def0c35f826a751e6314a8e6: Status 404 returned error can't find the container with id 99563b0bfdcd8ae53a02ea22728e14041933c6e6def0c35f826a751e6314a8e6
Sep 11 16:17:07.027472 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.027448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml"
Sep 11 16:17:07.034539 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.034516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cnvx5"
Sep 11 16:17:07.041522 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.041498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5fvg"
Sep 11 16:17:07.049197 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.049174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jhltr"
Sep 11 16:17:07.286401 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.286301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:07.286401 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.286387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:07.286672 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.286507    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.286672 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.286572    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.286554839 +0000 UTC m=+3.967659272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.286933 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.286894    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.287081 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.286984    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.286953448 +0000 UTC m=+3.968057859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.387121 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.387089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:07.387279 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.387230    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:07.387279 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.387247    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:07.387279 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.387256    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.387415 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:07.387309    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.387295399 +0000 UTC m=+4.068399806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.708021 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.707907    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:12:05 +0000 UTC" deadline="2028-03-03 17:33:16.506973333 +0000 UTC"
Sep 11 16:17:07.708021 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.707961    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12937h16m8.799016462s"
Sep 11 16:17:07.944203 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.944161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qm7s8" event={"ID":"7d65744c-640a-47cf-bd5d-966db47417af","Type":"ContainerStarted","Data":"4c36a09cd121f3a2f65053a5b68da1e194c0a607bbdb525077e794116e2a7e23"}
Sep 11 16:17:07.945455 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.945414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wl27" event={"ID":"b264b7cd-f54d-4f10-b644-22fc59c4ab43","Type":"ContainerStarted","Data":"c297a7f140a05d5249fcf0da7e4ecbfb61af74a484ce3a210779fd5cceab5cc1"}
Sep 11 16:17:07.947095 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.947054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnqfb" event={"ID":"3da9490e-eee0-48dc-b34e-d00efe02064f","Type":"ContainerStarted","Data":"8ac892e6101efaed75e96ca19a58419a43944ae5ca108d97b8a514a4f8a3f4c8"}
Sep 11 16:17:07.948469 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.948437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"99563b0bfdcd8ae53a02ea22728e14041933c6e6def0c35f826a751e6314a8e6"}
Sep 11 16:17:07.949746 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:07.949682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5b6cb" event={"ID":"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3","Type":"ContainerStarted","Data":"f21bc3aac38e207fc907c57861eb2eb3fe7fc74c4741dcfba5e3933ac341e43f"}
Sep 11 16:17:08.139460 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:08.139269    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63dce7a7_e9d6_4880_8c61_74ddcd4dea50.slice/crio-4d7216c486067d5c49882dcbb885bed47e29e51a1644f3c2a5cdb14ede7f5321 WatchSource:0}: Error finding container 4d7216c486067d5c49882dcbb885bed47e29e51a1644f3c2a5cdb14ede7f5321: Status 404 returned error can't find the container with id 4d7216c486067d5c49882dcbb885bed47e29e51a1644f3c2a5cdb14ede7f5321
Sep 11 16:17:08.140695 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:08.140668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9c8529_29e0_4e00_b061_becbd28e2f1b.slice/crio-5e1207e6625d55e3333db7bc25d98348236b862ee9339f4a867a570f8dd571eb WatchSource:0}: Error finding container 5e1207e6625d55e3333db7bc25d98348236b862ee9339f4a867a570f8dd571eb: Status 404 returned error can't find the container with id 5e1207e6625d55e3333db7bc25d98348236b862ee9339f4a867a570f8dd571eb
Sep 11 16:17:08.143540 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:08.143430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c23413_8a02_469b_bf68_39ddd5729968.slice/crio-fe9aec777e0c9dbee7414a86e4367fdf32ac4d5b3d5d8f5fc173485238c7e3be WatchSource:0}: Error finding container fe9aec777e0c9dbee7414a86e4367fdf32ac4d5b3d5d8f5fc173485238c7e3be: Status 404 returned error can't find the container with id fe9aec777e0c9dbee7414a86e4367fdf32ac4d5b3d5d8f5fc173485238c7e3be
Sep 11 16:17:08.144328 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:08.144284    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda073bb01_4ab4_4f26_aa5d_3f6eca7c2ae4.slice/crio-79b76461e4026181b9814e1430a1f7500d8a120be094e4db7d7a99ce74734ef0 WatchSource:0}: Error finding container 79b76461e4026181b9814e1430a1f7500d8a120be094e4db7d7a99ce74734ef0: Status 404 returned error can't find the container with id 79b76461e4026181b9814e1430a1f7500d8a120be094e4db7d7a99ce74734ef0
Sep 11 16:17:08.295184 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.295136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:08.295365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.295221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:08.295436 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.295382    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:08.295489 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.295446    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:17:10.295426885 +0000 UTC m=+5.976531302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:08.295937 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.295860    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:08.296022 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.295982    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:10.295964661 +0000 UTC m=+5.977069066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:08.315465 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.315269    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:08.396450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.396360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:08.396617 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.396547    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:08.396617 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.396573    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:08.396617 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.396586    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.396801 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.396650    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:10.396629406 +0000 UTC m=+6.077733811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.937237 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.937160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:08.938010 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.937303    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:08.938010 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.937686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:08.938010 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.937779    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:08.938010 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.937963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:08.938221 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:08.938037    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:08.970567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.970531    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f5c5a235530abfb8b7c3ed471e06b88" containerID="592827a8a5621747beb5db12dd30a52dab28694c8e5848fd420326e90654dee5" exitCode=0
Sep 11 16:17:08.970726 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.970626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal" event={"ID":"4f5c5a235530abfb8b7c3ed471e06b88","Type":"ContainerDied","Data":"592827a8a5621747beb5db12dd30a52dab28694c8e5848fd420326e90654dee5"}
Sep 11 16:17:08.982296 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.982259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal" event={"ID":"20e5ec496ea128525a2252a4ae0e8d2f","Type":"ContainerStarted","Data":"34c27735b9c8cd5bc547bc7cdaace21305de0e389b93838cdff5b04f6a50f83f"}
Sep 11 16:17:08.995127 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:08.995071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerStarted","Data":"fe9aec777e0c9dbee7414a86e4367fdf32ac4d5b3d5d8f5fc173485238c7e3be"}
Sep 11 16:17:09.023251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:09.021773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5fvg" event={"ID":"1e9c8529-29e0-4e00-b061-becbd28e2f1b","Type":"ContainerStarted","Data":"5e1207e6625d55e3333db7bc25d98348236b862ee9339f4a867a570f8dd571eb"}
Sep 11 16:17:09.035007 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:09.034972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml" event={"ID":"63dce7a7-e9d6-4880-8c61-74ddcd4dea50","Type":"ContainerStarted","Data":"4d7216c486067d5c49882dcbb885bed47e29e51a1644f3c2a5cdb14ede7f5321"}
Sep 11 16:17:09.037816 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:09.037789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cnvx5" event={"ID":"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4","Type":"ContainerStarted","Data":"79b76461e4026181b9814e1430a1f7500d8a120be094e4db7d7a99ce74734ef0"}
Sep 11 16:17:10.052443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.052401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal" event={"ID":"4f5c5a235530abfb8b7c3ed471e06b88","Type":"ContainerStarted","Data":"521a67c2655540ccf8e7ef1d5cc130c4ac24c693d36126e165d78c4b10951c7b"}
Sep 11 16:17:10.069092 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.069023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-32.ec2.internal" podStartSLOduration=5.069002401 podStartE2EDuration="5.069002401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:09.01345873 +0000 UTC m=+4.694563156" watchObservedRunningTime="2026-09-11 16:17:10.069002401 +0000 UTC m=+5.750106828"
Sep 11 16:17:10.319210 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.319124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:10.319210 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.319196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:10.319427 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.319346    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:10.319427 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.319411    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:17:14.319391139 +0000 UTC m=+10.000495547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:10.319847 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.319808    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:10.319927 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.319876    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:14.319861259 +0000 UTC m=+10.000965665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:10.420033 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.419993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:10.420200 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.420153    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:10.420200 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.420173    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:10.420200 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.420185    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:10.420345 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.420239    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:14.420221359 +0000 UTC m=+10.101325763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:10.934452 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.934415    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:10.934620 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.934526    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:10.934845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.934816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:10.934937 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.934904    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:10.934992 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:10.934957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:10.935060 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:10.935009    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:12.937223 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:12.937146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:12.937223 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:12.937176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:12.937223 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:12.937152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:12.937798 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:12.937273    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:12.937798 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:12.937360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:12.937798 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:12.937434    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:14.357782 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:14.357734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:14.358262 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:14.357798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:14.358262 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.357919    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:14.358262 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.358006    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:22.357986002 +0000 UTC m=+18.039090407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:14.358262 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.357924    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:14.358262 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.358081    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:17:22.358063015 +0000 UTC m=+18.039167420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:14.458418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:14.458373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:14.458588 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.458528    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:14.458588 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.458543    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:14.458588 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.458555    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:14.458742 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.458611    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:22.458596965 +0000 UTC m=+18.139701370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:14.935256 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:14.935215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:14.935427 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.935324    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:14.935427 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:14.935411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:14.935590 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:14.935567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:14.935590 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.935572    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:14.935708 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:14.935682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:16.935302 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:16.935267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:16.935751 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:16.935326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:16.935751 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:16.935352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:16.935751 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:16.935422    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:16.935751 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:16.935473    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:16.935751 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:16.935590    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:18.934931 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:18.934895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:18.935397 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:18.934895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:18.935397 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:18.935007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:18.935397 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:18.934895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:18.935397 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:18.935102    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:18.935397 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:18.935195    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:20.934986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:20.934699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:20.935397 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:20.934712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:20.935397 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:20.935084    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:20.935397 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:20.934747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:20.935397 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:20.935154    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:20.935397 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:20.935216    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:22.413636 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:22.413598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:22.414081 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:22.413653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:22.414081 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.413764    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:22.414081 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.413766    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:22.414081 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.413817    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:17:38.413803536 +0000 UTC m=+34.094907938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:22.414081 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.413851    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:38.413840024 +0000 UTC m=+34.094944433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:22.515044 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:22.515007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:22.515217 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.515190    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:22.515217 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.515213    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:22.515351 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.515226    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:22.515351 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.515293    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:38.515272986 +0000 UTC m=+34.196377419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:22.934853 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:22.934790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:22.934853 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:22.934815    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:22.935078 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:22.934798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:22.935078 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.934936    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:22.935078 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.935018    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:22.935228 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:22.935114    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:24.935118 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:24.935084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:24.935561 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:24.935178    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:24.935561 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:24.935202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:24.935561 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:24.935256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:24.935561 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:24.935333    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:24.935561 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:24.935394    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:26.085443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.085156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"5a73fe86bcad52bb1f5051ce2159136524e8c276b2f248b232934eb3f1f120aa"}
Sep 11 16:17:26.086107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.085505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"be49151fe0c39eaa2e247a8ca8ab6dcbae04ac1e61898a041825dbbae373a60c"}
Sep 11 16:17:26.086107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.085562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"a2d65481acb5b92e3315a40d305207342ee9786054bd11a65a263187b77a0a6a"}
Sep 11 16:17:26.086107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.085610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"bdc00d3842eba18d03a98ac578d00b529a6025afb8da3ae33536c3d2a8006319"}
Sep 11 16:17:26.086107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.085661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"e4c926e396d7e718af825d9755d738fcbe54e9ae3d2564083586fd279db97474"}
Sep 11 16:17:26.086107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.085712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"762d31e733029695bb5d18770cc6711a670387ea63018874eae29d35a29f689b"}
Sep 11 16:17:26.087246 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.087223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5b6cb" event={"ID":"c764ae30-1eb4-43a7-a78e-bd975b3fa3e3","Type":"ContainerStarted","Data":"493a1153ef69700b90c77d8a475a3c08dbfcf06fe2f72a44f9fb9f781c0a67bf"}
Sep 11 16:17:26.090676 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.090647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cnvx5" event={"ID":"a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4","Type":"ContainerStarted","Data":"7bd04104e0371fecf1b86d860b5029d30a6a8a798ffb0472926fc5e149865737"}
Sep 11 16:17:26.092516 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.092482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qm7s8" event={"ID":"7d65744c-640a-47cf-bd5d-966db47417af","Type":"ContainerStarted","Data":"a614c87dc565094319bca405658679bc73b2ea7a4ded5f8492a78f7da3e8d16d"}
Sep 11 16:17:26.094106 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.094084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wl27" event={"ID":"b264b7cd-f54d-4f10-b644-22fc59c4ab43","Type":"ContainerStarted","Data":"7870a1716f3895884f780daed4613cf8b473b20173d8177ff2b299b85b85da23"}
Sep 11 16:17:26.098922 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.098893    2580 generic.go:356] "Generic (PLEG): container finished" podID="58c23413-8a02-469b-bf68-39ddd5729968" containerID="758f8d09232ae4f80c35e3e92251d8f739f29510b70db68391f9ddddc9d1a0d6" exitCode=0
Sep 11 16:17:26.099014 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.098971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerDied","Data":"758f8d09232ae4f80c35e3e92251d8f739f29510b70db68391f9ddddc9d1a0d6"}
Sep 11 16:17:26.100656 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.100635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5fvg" event={"ID":"1e9c8529-29e0-4e00-b061-becbd28e2f1b","Type":"ContainerStarted","Data":"1e1d16706ea0a5d5cb460da42394afd48f333dca1384e194d4c632ac1ad13d34"}
Sep 11 16:17:26.102616 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.102596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml" event={"ID":"63dce7a7-e9d6-4880-8c61-74ddcd4dea50","Type":"ContainerStarted","Data":"aa4845f27b34b426da8fcf78aa148c36a99490000db6d3a7004af41d17b58ee0"}
Sep 11 16:17:26.107184 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.107149    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-32.ec2.internal" podStartSLOduration=21.107124889 podStartE2EDuration="21.107124889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:10.069757749 +0000 UTC m=+5.750862173" watchObservedRunningTime="2026-09-11 16:17:26.107124889 +0000 UTC m=+21.788229291"
Sep 11 16:17:26.107465 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.107444    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5b6cb" podStartSLOduration=3.016621927 podStartE2EDuration="21.107436237s" totalImagesPullingTime="18.09081431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.107193118 +0000 UTC m=+21.788297543" watchObservedRunningTime="2026-09-11 16:17:26.107436237 +0000 UTC m=+21.788540661"
Sep 11 16:17:26.124347 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.124311    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2wl27" podStartSLOduration=3.058296878 podStartE2EDuration="21.124299618s" totalImagesPullingTime="18.06600274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.123737357 +0000 UTC m=+21.804841783" watchObservedRunningTime="2026-09-11 16:17:26.124299618 +0000 UTC m=+21.805404043"
Sep 11 16:17:26.141907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.141870    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cnvx5" podStartSLOduration=4.217935656 podStartE2EDuration="21.141857636s" totalImagesPullingTime="16.92392198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.141727328 +0000 UTC m=+21.822831755" watchObservedRunningTime="2026-09-11 16:17:26.141857636 +0000 UTC m=+21.822962060"
Sep 11 16:17:26.163284 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.163244    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qm7s8" podStartSLOduration=3.123801451 podStartE2EDuration="21.163231959s" totalImagesPullingTime="18.039430508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.163050044 +0000 UTC m=+21.844154470" watchObservedRunningTime="2026-09-11 16:17:26.163231959 +0000 UTC m=+21.844336425"
Sep 11 16:17:26.188928 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.188880    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v5fvg" podStartSLOduration=8.390443954 podStartE2EDuration="21.188863181s" totalImagesPullingTime="12.798419227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.188626839 +0000 UTC m=+21.869731264" watchObservedRunningTime="2026-09-11 16:17:26.188863181 +0000 UTC m=+21.869967608"
Sep 11 16:17:26.258587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.258558    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:26.340330 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.340241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:26.479402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.479378    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:17:26.734150 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.734033    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:17:26.479393858Z","UUID":"ac1d081b-7284-46c8-b116-0a29c749162b","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:17:26.735804 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.735787    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:17:26.735937 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.735813    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:17:26.934855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.934807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:26.935028 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.934872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:26.935028 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:26.934812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:26.935028 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:26.934952    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:26.935197 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:26.935030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:26.935197 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:26.935106    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:27.108368 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:27.108144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml" event={"ID":"63dce7a7-e9d6-4880-8c61-74ddcd4dea50","Type":"ContainerStarted","Data":"d582a8e220cb8c334d10306b99c1ed3b7ff6e224dd581426a4b5dfd8f996da42"}
Sep 11 16:17:27.110811 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:27.110780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnqfb" event={"ID":"3da9490e-eee0-48dc-b34e-d00efe02064f","Type":"ContainerStarted","Data":"5ace9349b36b71c2b9beef513e0b161589d85cd2844666a4b55bcc9f9a86ff12"}
Sep 11 16:17:27.128667 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:27.128614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cnqfb" podStartSLOduration=4.051274047 podStartE2EDuration="22.128587447s" totalImagesPullingTime="18.0773134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:27.128250128 +0000 UTC m=+22.809354554" watchObservedRunningTime="2026-09-11 16:17:27.128587447 +0000 UTC m=+22.809691873"
Sep 11 16:17:27.251394 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:27.251360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:27.311714 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:27.311628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.116525 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.116488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml" event={"ID":"63dce7a7-e9d6-4880-8c61-74ddcd4dea50","Type":"ContainerStarted","Data":"d5b33e32e5cbb978698a2fdd637160eb9bb6e71ce63666631e9c4194ecf386cc"}
Sep 11 16:17:28.120352 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.120308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"e9d63b96971af8bd2dfceb3da7b676c13e95456930e25e5d5fdd483e33ff3330"}
Sep 11 16:17:28.136901 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.136854    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glnml" podStartSLOduration=3.998734277 podStartE2EDuration="23.136819881s" totalImagesPullingTime="19.138085604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:28.136152725 +0000 UTC m=+23.817257150" watchObservedRunningTime="2026-09-11 16:17:28.136819881 +0000 UTC m=+23.817924308"
Sep 11 16:17:28.500410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.500374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:28.500996 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.500968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:28.679205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.679172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:28.679770 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.679748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2wl27"
Sep 11 16:17:28.756507 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.756442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.934985 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.934953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:28.935161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.935000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:28.935161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:28.934967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:28.935161 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:28.935083    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:28.935325 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:28.935181    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:28.935325 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:28.935273    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:29.250922 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:29.250884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:29.321945 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:29.321910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:30.934934 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:30.934723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:30.935278 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:30.934727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:30.935278 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:30.935015    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:30.935278 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:30.935122    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:30.935278 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:30.934764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:30.935278 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:30.935210    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:31.128629 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:31.128592    2580 generic.go:356] "Generic (PLEG): container finished" podID="58c23413-8a02-469b-bf68-39ddd5729968" containerID="ba74272e2f6a1dc213ff80b37de83ca9bf6af5f3b8c76e1e16dd96dbcd28aee5" exitCode=0
Sep 11 16:17:31.128780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:31.128673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerDied","Data":"ba74272e2f6a1dc213ff80b37de83ca9bf6af5f3b8c76e1e16dd96dbcd28aee5"}
Sep 11 16:17:31.132524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:31.132504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" event={"ID":"d91b4ccf-9425-4777-9bac-32be72517058","Type":"ContainerStarted","Data":"c3ebf10406635182fb0f160c0c68572ef25f977a83603275d0035703857665c4"}
Sep 11 16:17:31.132768 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:31.132756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:31.147434 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:31.147416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:31.179855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:31.179798    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g" podStartSLOduration=7.744233147 podStartE2EDuration="26.179787334s" totalImagesPullingTime="18.435554187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:31.179225559 +0000 UTC m=+26.860329981" watchObservedRunningTime="2026-09-11 16:17:31.179787334 +0000 UTC m=+26.860891760"
Sep 11 16:17:32.093858 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.093771    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhpp4"]
Sep 11 16:17:32.094605 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.093900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:32.094605 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:32.094010    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:32.096764 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.096739    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdf6j"]
Sep 11 16:17:32.096910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.096854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:32.096981 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:32.096947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:32.097413 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.097390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gmprr"]
Sep 11 16:17:32.097513 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.097475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:32.097586 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:32.097565    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:32.137285 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.137254    2580 generic.go:356] "Generic (PLEG): container finished" podID="58c23413-8a02-469b-bf68-39ddd5729968" containerID="73a3d42744ea3f7e39ff370856bee3fefeb5b0a28fe9dede4d79c4f8daff0dbd" exitCode=0
Sep 11 16:17:32.137409 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.137330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerDied","Data":"73a3d42744ea3f7e39ff370856bee3fefeb5b0a28fe9dede4d79c4f8daff0dbd"}
Sep 11 16:17:32.137788 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.137769    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:32.137927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.137839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:32.153496 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.153475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:17:32.889908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:32.889876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:33.142399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.142333    2580 generic.go:356] "Generic (PLEG): container finished" podID="58c23413-8a02-469b-bf68-39ddd5729968" containerID="c573fe85fd0df27ddb15ce487bbae6f43d172c18d09f5deba4bece3640170d46" exitCode=0
Sep 11 16:17:33.142722 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.142412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerDied","Data":"c573fe85fd0df27ddb15ce487bbae6f43d172c18d09f5deba4bece3640170d46"}
Sep 11 16:17:33.486483 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.486225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:33.549367 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.549332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:33.886298 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.886250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:33.935232 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.935208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:33.935400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.935211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:33.935400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.935211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:33.935494 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:33.935416    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:33.935494 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:33.935300    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:33.935494 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:33.935467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:33.980840 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:33.980804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:34.045245 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:34.045209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:34.224153 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:34.224033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:34.301869 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:34.301810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:35.833196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:35.833121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:35.934395 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:35.934366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:35.934395 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:35.934403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:35.934633 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:35.934485    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:35.934633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:35.934508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:35.934633 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:35.934579    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:35.934793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:35.934659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:35.934793 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:35.934699    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:36.164450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:36.164415    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5fvg_1e9c8529-29e0-4e00-b061-becbd28e2f1b/dns-node-resolver/0.log"
Sep 11 16:17:36.766596 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:36.766572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qm7s8_7d65744c-640a-47cf-bd5d-966db47417af/node-ca/0.log"
Sep 11 16:17:37.035841 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.035713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.346154 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.346067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.562785 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.562752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.667295 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.667256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.775370 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.775340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.885346 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.885310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.934406 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.934336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:37.934538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.934337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:37.934538 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:37.934439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:37.934538 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:37.934518    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:37.934538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.934336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:37.934661 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:37.934592    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:37.986155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:37.986124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:38.092184 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.092148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:38.157740 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.157710    2580 generic.go:356] "Generic (PLEG): container finished" podID="58c23413-8a02-469b-bf68-39ddd5729968" containerID="df94b7ab92d6d48ccd8ae5b4f25f20a4d615266fff7e5f092e231fd38342b3e6" exitCode=0
Sep 11 16:17:38.157904 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.157750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerDied","Data":"df94b7ab92d6d48ccd8ae5b4f25f20a4d615266fff7e5f092e231fd38342b3e6"}
Sep 11 16:17:38.281207 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.281183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:38.433987 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.433958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:38.434109 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.433999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:38.434109 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.434096    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:38.434192 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.434154    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret podName:1c159bf6-5095-4cf6-b9fc-0935e2fd00ca nodeName:}" failed. No retries permitted until 2026-09-11 16:18:10.434139508 +0000 UTC m=+66.115243915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret") pod "global-pull-secret-syncer-gmprr" (UID: "1c159bf6-5095-4cf6-b9fc-0935e2fd00ca") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:38.434192 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.434096    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:38.434264 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.434227    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs podName:8cd81b90-21f6-4343-99c5-3d746e3c480e nodeName:}" failed. No retries permitted until 2026-09-11 16:18:10.434214304 +0000 UTC m=+66.115318706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs") pod "network-metrics-daemon-lhpp4" (UID: "8cd81b90-21f6-4343-99c5-3d746e3c480e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:38.535187 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:38.535119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:38.535306 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.535262    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:38.535306 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.535281    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:38.535306 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.535290    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hgdvr for pod openshift-network-diagnostics/network-check-target-bdf6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:38.535416 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:38.535334    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr podName:c0eb169c-3c31-48f2-a973-53f0aacced29 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:10.535321531 +0000 UTC m=+66.216425934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgdvr" (UniqueName: "kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr") pod "network-check-target-bdf6j" (UID: "c0eb169c-3c31-48f2-a973-53f0aacced29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:39.163406 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.163372    2580 generic.go:356] "Generic (PLEG): container finished" podID="58c23413-8a02-469b-bf68-39ddd5729968" containerID="d207b4ec71e7da531e681023e1063d12eb5c347ca8b4b14556cf762f6aa548fe" exitCode=0
Sep 11 16:17:39.163774 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.163453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerDied","Data":"d207b4ec71e7da531e681023e1063d12eb5c347ca8b4b14556cf762f6aa548fe"}
Sep 11 16:17:39.264154 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.264000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:39.652382 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.652351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:39.858033 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.857999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:39.935208 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.935137    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:39.935208 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.935156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:39.935375 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:39.935143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:39.935375 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:39.935225    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:39.935375 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:39.935295    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:39.935375 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:39.935360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:40.055400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:40.055361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:40.169414 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:40.169384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jhltr" event={"ID":"58c23413-8a02-469b-bf68-39ddd5729968","Type":"ContainerStarted","Data":"e2714ec4b3c78201c7a994c56c3ed667bf89a1a2e2132c4ad7755395365a42f2"}
Sep 11 16:17:40.195309 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:40.195237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:40.195419 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:40.195384    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jhltr" podStartSLOduration=8.45618355 podStartE2EDuration="35.195368333s" totalImagesPullingTime="26.739184783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:40.194671976 +0000 UTC m=+35.875776402" watchObservedRunningTime="2026-09-11 16:17:40.195368333 +0000 UTC m=+35.876472758"
Sep 11 16:17:40.454740 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:40.454657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.047457 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.047427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.147170 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.147139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.250185 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.250151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.437621 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.437585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.655389 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.655351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.850534 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.850465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:41.934650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.934622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:41.934781 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.934711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:41.934781 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:41.934720    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:41.934781 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:41.934738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:41.934899 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:41.934811    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:41.934936 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:41.934899    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:42.255417 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:42.255380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:42.314708 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:42.314679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:42.452103 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:42.452072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.058940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.058908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.246025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.245991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.317443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.317371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.857156 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.857129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.923143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.923113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.934952 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.934934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:43.935059 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.934938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:43.935059 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:43.935020    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:43.935137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:43.934942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:43.935137 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:43.935091    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:43.935199 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:43.935167    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:44.242267 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:44.242229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:44.305712 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:44.305684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:44.456159 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:44.456123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:45.051420 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:45.051386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:45.654113 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:45.654076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:45.857161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:45.857131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:45.935245 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:45.935172    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:45.935245 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:45.935188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:45.935429 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:45.935280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:45.935429 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:45.935281    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:45.935429 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:45.935342    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:45.935549 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:45.935421    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:46.258951 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:46.258865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:46.453451 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:46.453423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:46.850098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:46.850064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:47.679133 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:47.679090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:47.873870 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:47.873841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:47.934946 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:47.934881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:47.934946 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:47.934892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:47.934946 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:47.934908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:47.935150 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:47.934968    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:47.935150 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:47.935035    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:47.935150 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:47.935085    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:48.062383 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:48.062350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:48.265788 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:48.265702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:48.349929 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:48.349893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:49.935097 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:49.935067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:49.935479 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:49.935149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:49.935479 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:49.935156    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:49.935479 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:49.935223    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:49.935479 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:49.935233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:49.935479 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:49.935323    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:51.934608 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:51.934576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:51.935084 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:51.934574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:51.935084 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:51.934673    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:51.935084 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:51.934582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:51.935084 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:51.934741    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:51.935084 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:51.934868    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:53.935315 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:53.935280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:53.935315 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:53.935317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:53.935782 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:53.935281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:53.935782 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:53.935397    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:53.935782 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:53.935462    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:53.935782 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:53.935511    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:55.934402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:55.934370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:55.934402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:55.934398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:55.934779 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:55.934373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:55.934779 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:55.934461    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:55.934779 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:55.934557    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:55.934779 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:55.934615    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:57.044935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.044899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:57.138683 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.138650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:57.239023 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.238966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:57.339631 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.339557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:57.934943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.934905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:57.935113 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.934918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:57.935113 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:57.935026    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhpp4" podUID="8cd81b90-21f6-4343-99c5-3d746e3c480e"
Sep 11 16:17:57.935113 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:57.934919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:57.935113 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:57.935083    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdf6j" podUID="c0eb169c-3c31-48f2-a973-53f0aacced29"
Sep 11 16:17:57.935252 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:17:57.935165    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gmprr" podUID="1c159bf6-5095-4cf6-b9fc-0935e2fd00ca"
Sep 11 16:17:58.077315 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.077235    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-32.ec2.internal" event="NodeReady"
Sep 11 16:17:58.077657 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.077341    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:17:58.123817 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.123783    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76485bf8fb-wn22k"]
Sep 11 16:17:58.125534 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.125517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.128642 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.128619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 16:17:58.128733 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.128692    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 16:17:58.129024 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.128997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 16:17:58.129122 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.129085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cx76c\""
Sep 11 16:17:58.135716 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.135699    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 16:17:58.137884 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.137867    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fnwsz"]
Sep 11 16:17:58.139342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.139327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.142151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.142132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:17:58.142682 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.142639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:17:58.143206 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.143168    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:17:58.143584 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.143555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hnfgd\""
Sep 11 16:17:58.144154 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.144137    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m7dmz"]
Sep 11 16:17:58.145988 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.145974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.151389 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.151343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:17:58.151693 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.151672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76485bf8fb-wn22k"]
Sep 11 16:17:58.151788 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.151675    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:17:58.151788 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.151746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gnx8w\""
Sep 11 16:17:58.152000 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.151986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:17:58.152136 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.152119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:17:58.155433 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.155416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnwsz"]
Sep 11 16:17:58.165910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.165884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7dmz"]
Sep 11 16:17:58.177499 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-trusted-ca\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177499 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82e9b905-460b-4c58-b2ae-c64e8c4fa5d3-cert\") pod \"ingress-canary-fnwsz\" (UID: \"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3\") " pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.177646 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56f5fd17-90aa-42e8-9989-6cfa74f86449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.177646 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzr4c\" (UniqueName: \"kubernetes.io/projected/56f5fd17-90aa-42e8-9989-6cfa74f86449-kube-api-access-nzr4c\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.177646 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-installation-pull-secrets\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177742 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56f5fd17-90aa-42e8-9989-6cfa74f86449-crio-socket\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.177742 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-registry-tls\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177742 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-ca-trust-extracted\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177851 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56f5fd17-90aa-42e8-9989-6cfa74f86449-data-volume\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.177851 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-bound-sa-token\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177851 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56f5fd17-90aa-42e8-9989-6cfa74f86449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.177948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5sh\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-kube-api-access-2f5sh\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hlr\" (UniqueName: \"kubernetes.io/projected/82e9b905-460b-4c58-b2ae-c64e8c4fa5d3-kube-api-access-j9hlr\") pod \"ingress-canary-fnwsz\" (UID: \"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3\") " pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.177948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-registry-certificates\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.177948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.177923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-image-registry-private-configuration\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.229634 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.229599    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jzs9s"]
Sep 11 16:17:58.231276 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.231261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.234949 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.234927    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lsfrm\""
Sep 11 16:17:58.235051 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.234969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:17:58.235322 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.235303    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:17:58.251989 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.251963    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzs9s"]
Sep 11 16:17:58.278563 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5sh\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-kube-api-access-2f5sh\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.278563 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hlr\" (UniqueName: \"kubernetes.io/projected/82e9b905-460b-4c58-b2ae-c64e8c4fa5d3-kube-api-access-j9hlr\") pod \"ingress-canary-fnwsz\" (UID: \"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3\") " pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.278732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc682-6484-4657-9d87-e47fa86e088a-tmp-dir\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.278732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-registry-certificates\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.278732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-image-registry-private-configuration\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.278732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-trusted-ca\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.278732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82e9b905-460b-4c58-b2ae-c64e8c4fa5d3-cert\") pod \"ingress-canary-fnwsz\" (UID: \"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3\") " pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.278732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56f5fd17-90aa-42e8-9989-6cfa74f86449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzr4c\" (UniqueName: \"kubernetes.io/projected/56f5fd17-90aa-42e8-9989-6cfa74f86449-kube-api-access-nzr4c\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-installation-pull-secrets\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56f5fd17-90aa-42e8-9989-6cfa74f86449-crio-socket\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-registry-tls\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-ca-trust-extracted\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bfc682-6484-4657-9d87-e47fa86e088a-config-volume\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56f5fd17-90aa-42e8-9989-6cfa74f86449-data-volume\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56f5fd17-90aa-42e8-9989-6cfa74f86449-crio-socket\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.279042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.278946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4bfc682-6484-4657-9d87-e47fa86e088a-metrics-tls\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.279501 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.279350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hs4\" (UniqueName: \"kubernetes.io/projected/b4bfc682-6484-4657-9d87-e47fa86e088a-kube-api-access-c4hs4\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.279501 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.279422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-bound-sa-token\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.279501 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.279446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56f5fd17-90aa-42e8-9989-6cfa74f86449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.279630 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.279549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-registry-certificates\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.279805 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.279784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-ca-trust-extracted\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.279894 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.279864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-trusted-ca\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.280152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.280126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56f5fd17-90aa-42e8-9989-6cfa74f86449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.280422 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.280403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56f5fd17-90aa-42e8-9989-6cfa74f86449-data-volume\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.282816 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.282792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-image-registry-private-configuration\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.282955 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.282936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-installation-pull-secrets\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.283034 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.282974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56f5fd17-90aa-42e8-9989-6cfa74f86449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.283132 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.283115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82e9b905-460b-4c58-b2ae-c64e8c4fa5d3-cert\") pod \"ingress-canary-fnwsz\" (UID: \"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3\") " pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.283426 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.283409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-registry-tls\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.288145 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.288121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hlr\" (UniqueName: \"kubernetes.io/projected/82e9b905-460b-4c58-b2ae-c64e8c4fa5d3-kube-api-access-j9hlr\") pod \"ingress-canary-fnwsz\" (UID: \"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3\") " pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.288299 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.288281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzr4c\" (UniqueName: \"kubernetes.io/projected/56f5fd17-90aa-42e8-9989-6cfa74f86449-kube-api-access-nzr4c\") pod \"insights-runtime-extractor-m7dmz\" (UID: \"56f5fd17-90aa-42e8-9989-6cfa74f86449\") " pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.288518 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.288504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-bound-sa-token\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.289198 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.289183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5sh\" (UniqueName: \"kubernetes.io/projected/a0abb8f7-a72a-45df-8a45-9c0adc0b220e-kube-api-access-2f5sh\") pod \"image-registry-76485bf8fb-wn22k\" (UID: \"a0abb8f7-a72a-45df-8a45-9c0adc0b220e\") " pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.376331 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.376294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:58.382358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.380614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc682-6484-4657-9d87-e47fa86e088a-tmp-dir\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.382358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.380730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bfc682-6484-4657-9d87-e47fa86e088a-config-volume\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.382358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.380764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4bfc682-6484-4657-9d87-e47fa86e088a-metrics-tls\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.382358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.380791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hs4\" (UniqueName: \"kubernetes.io/projected/b4bfc682-6484-4657-9d87-e47fa86e088a-kube-api-access-c4hs4\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.382358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.381434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bfc682-6484-4657-9d87-e47fa86e088a-tmp-dir\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.382358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.381924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bfc682-6484-4657-9d87-e47fa86e088a-config-volume\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.384695 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.384669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4bfc682-6484-4657-9d87-e47fa86e088a-metrics-tls\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.390348 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.390323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hs4\" (UniqueName: \"kubernetes.io/projected/b4bfc682-6484-4657-9d87-e47fa86e088a-kube-api-access-c4hs4\") pod \"dns-default-jzs9s\" (UID: \"b4bfc682-6484-4657-9d87-e47fa86e088a\") " pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.435220 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.435195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:58.447699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.447680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnwsz"
Sep 11 16:17:58.454366 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.454345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7dmz"
Sep 11 16:17:58.476704 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.476670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:58.539162 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.539138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:17:58.724720 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.724693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnwsz"]
Sep 11 16:17:58.728729 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.728699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76485bf8fb-wn22k"]
Sep 11 16:17:58.731613 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:58.731591    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0abb8f7_a72a_45df_8a45_9c0adc0b220e.slice/crio-304f32a338a41dad32fb214fe63e0e528f7d8b4c35cbfc02760bff46641ae640 WatchSource:0}: Error finding container 304f32a338a41dad32fb214fe63e0e528f7d8b4c35cbfc02760bff46641ae640: Status 404 returned error can't find the container with id 304f32a338a41dad32fb214fe63e0e528f7d8b4c35cbfc02760bff46641ae640
Sep 11 16:17:58.746112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.746083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7dmz"]
Sep 11 16:17:58.749482 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:58.749453    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f5fd17_90aa_42e8_9989_6cfa74f86449.slice/crio-817cb8e61d54e818667122daab2df49ed1792054d065f68f999dff596db52a1d WatchSource:0}: Error finding container 817cb8e61d54e818667122daab2df49ed1792054d065f68f999dff596db52a1d: Status 404 returned error can't find the container with id 817cb8e61d54e818667122daab2df49ed1792054d065f68f999dff596db52a1d
Sep 11 16:17:58.810808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:58.810783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzs9s"]
Sep 11 16:17:58.813616 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:17:58.813593    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bfc682_6484_4657_9d87_e47fa86e088a.slice/crio-dca559280d4a450b642248b778d0dec16b273dbea70f0a7afd1de5c33cd061f5 WatchSource:0}: Error finding container dca559280d4a450b642248b778d0dec16b273dbea70f0a7afd1de5c33cd061f5: Status 404 returned error can't find the container with id dca559280d4a450b642248b778d0dec16b273dbea70f0a7afd1de5c33cd061f5
Sep 11 16:17:59.211902 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.211836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnwsz" event={"ID":"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3","Type":"ContainerStarted","Data":"1fbcc9640648191a13b889d0c7d1a6cddf2792bdbf9a7d3ad6c9f83dc9863436"}
Sep 11 16:17:59.213898 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.213868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k" event={"ID":"a0abb8f7-a72a-45df-8a45-9c0adc0b220e","Type":"ContainerStarted","Data":"d00729f5669a714ee6b247abf81efed39ba0f76be5df2316d2d6a179a3383432"}
Sep 11 16:17:59.214050 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.213920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k" event={"ID":"a0abb8f7-a72a-45df-8a45-9c0adc0b220e","Type":"ContainerStarted","Data":"304f32a338a41dad32fb214fe63e0e528f7d8b4c35cbfc02760bff46641ae640"}
Sep 11 16:17:59.214123 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.214093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:17:59.215871 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.215847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7dmz" event={"ID":"56f5fd17-90aa-42e8-9989-6cfa74f86449","Type":"ContainerStarted","Data":"99121c90e443a086473d8ddb6989bff3b5f06a816d5e939d9eea042694909ccb"}
Sep 11 16:17:59.215991 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.215899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7dmz" event={"ID":"56f5fd17-90aa-42e8-9989-6cfa74f86449","Type":"ContainerStarted","Data":"817cb8e61d54e818667122daab2df49ed1792054d065f68f999dff596db52a1d"}
Sep 11 16:17:59.217390 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.217328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzs9s" event={"ID":"b4bfc682-6484-4657-9d87-e47fa86e088a","Type":"ContainerStarted","Data":"dca559280d4a450b642248b778d0dec16b273dbea70f0a7afd1de5c33cd061f5"}
Sep 11 16:17:59.935404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.935362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:17:59.935404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.935383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:17:59.935653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.935363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:17:59.938591 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.938568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:59.938704 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.938568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:59.940189 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.940171    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:59.940316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.940236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:59.940316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.940261    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mk9m2\""
Sep 11 16:17:59.940442 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:17:59.940350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn9xf\""
Sep 11 16:18:00.222662 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:00.222580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7dmz" event={"ID":"56f5fd17-90aa-42e8-9989-6cfa74f86449","Type":"ContainerStarted","Data":"83028175212a24e4aba097fec0685a6e45482017bffd492839c037ee37fc090e"}
Sep 11 16:18:02.821999 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:02.821903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:02.904094 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:02.904059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.006629 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.006593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.112793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.112760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.210577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.210546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.233935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.233903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7dmz" event={"ID":"56f5fd17-90aa-42e8-9989-6cfa74f86449","Type":"ContainerStarted","Data":"a8b53f808538ae091007ff9925dca0be4906f44fd6272c770a5f684372de7657"}
Sep 11 16:18:03.236884 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.236395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzs9s" event={"ID":"b4bfc682-6484-4657-9d87-e47fa86e088a","Type":"ContainerStarted","Data":"f6a965bac3ea0f811650815b617c87b734c86bb3c1989772fa350b46ef1c8abf"}
Sep 11 16:18:03.236884 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.236464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzs9s" event={"ID":"b4bfc682-6484-4657-9d87-e47fa86e088a","Type":"ContainerStarted","Data":"55c8fe1c9eccececdfe81daf7643e4f5cd92c58f7bf5916702d50101c02af640"}
Sep 11 16:18:03.236884 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.236546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:18:03.238536 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.238514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnwsz" event={"ID":"82e9b905-460b-4c58-b2ae-c64e8c4fa5d3","Type":"ContainerStarted","Data":"c06127b6116e0edb18554779638155d63069192ec0bad446db91c615f73bfc5b"}
Sep 11 16:18:03.252837 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.252781    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k" podStartSLOduration=6.252767946 podStartE2EDuration="6.252767946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:59.238609798 +0000 UTC m=+54.919714224" watchObservedRunningTime="2026-09-11 16:18:03.252767946 +0000 UTC m=+58.933872372"
Sep 11 16:18:03.272046 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.272005    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m7dmz" podStartSLOduration=1.824589915 podStartE2EDuration="5.271991727s" totalImagesPullingTime="3.447401812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:03.253693054 +0000 UTC m=+58.934797478" watchObservedRunningTime="2026-09-11 16:18:03.271991727 +0000 UTC m=+58.953096148"
Sep 11 16:18:03.272127 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.272089    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jzs9s" podStartSLOduration=1.7723098419999999 podStartE2EDuration="5.272085406s" totalImagesPullingTime="3.499775564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:03.271556753 +0000 UTC m=+58.952661261" watchObservedRunningTime="2026-09-11 16:18:03.272085406 +0000 UTC m=+58.953189830"
Sep 11 16:18:03.288586 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.288539    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fnwsz" podStartSLOduration=1.70107039 podStartE2EDuration="5.288517713s" totalImagesPullingTime="3.587447323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:03.287987804 +0000 UTC m=+58.969092229" watchObservedRunningTime="2026-09-11 16:18:03.288517713 +0000 UTC m=+58.969622142"
Sep 11 16:18:03.782949 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.782911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.850455 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:03.850429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:04.155423 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:04.155397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s9f5g"
Sep 11 16:18:04.189701 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:04.189671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:04.290646 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:04.290609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:04.993905 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:04.993868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.073714 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:05.073686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.402924 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:05.402894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.499141 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:05.499103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.355415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.355376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.451246 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.451213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.545088 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.545052    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"]
Sep 11 16:18:06.547101 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.547074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.547877 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.547859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:06.552553 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.552473    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 16:18:06.552642 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.552564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xwbz9\""
Sep 11 16:18:06.566973 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.566951    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"]
Sep 11 16:18:06.653042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.653017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb0d6381-a374-4372-b711-9ebe6700d403-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5gxx\" (UID: \"bb0d6381-a374-4372-b711-9ebe6700d403\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:06.720510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.720476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.754177 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.754151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb0d6381-a374-4372-b711-9ebe6700d403-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5gxx\" (UID: \"bb0d6381-a374-4372-b711-9ebe6700d403\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:06.756434 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.756415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb0d6381-a374-4372-b711-9ebe6700d403-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5gxx\" (UID: \"bb0d6381-a374-4372-b711-9ebe6700d403\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:06.858273 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:06.858238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:07.125009 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:07.124974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"]
Sep 11 16:18:07.127692 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:07.127664    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0d6381_a374_4372_b711_9ebe6700d403.slice/crio-38500b500f78062a307c5b0018bcad384b9d3dbf2a44abb2bb318b5243c15aa0 WatchSource:0}: Error finding container 38500b500f78062a307c5b0018bcad384b9d3dbf2a44abb2bb318b5243c15aa0: Status 404 returned error can't find the container with id 38500b500f78062a307c5b0018bcad384b9d3dbf2a44abb2bb318b5243c15aa0
Sep 11 16:18:07.252962 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:07.252878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx" event={"ID":"bb0d6381-a374-4372-b711-9ebe6700d403","Type":"ContainerStarted","Data":"38500b500f78062a307c5b0018bcad384b9d3dbf2a44abb2bb318b5243c15aa0"}
Sep 11 16:18:09.259983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.259947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx" event={"ID":"bb0d6381-a374-4372-b711-9ebe6700d403","Type":"ContainerStarted","Data":"5a372047e2b45526acc255b96a1eb89a867a054ee7cf5386f4012fcc7b86b3d2"}
Sep 11 16:18:09.260454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.260152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:09.264719 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.264693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx"
Sep 11 16:18:09.279507 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.279463    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5gxx" podStartSLOduration=1.696321494 podStartE2EDuration="3.279447371s" totalImagesPullingTime="1.583125877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:09.278685846 +0000 UTC m=+64.959790272" watchObservedRunningTime="2026-09-11 16:18:09.279447371 +0000 UTC m=+64.960551796"
Sep 11 16:18:09.592590 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.592470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.630280 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.630249    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"]
Sep 11 16:18:09.632110 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.632095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.635143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.635121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:18:09.635257 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.635121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:18:09.636361 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.636343    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 16:18:09.636458 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.636360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rztmd\""
Sep 11 16:18:09.636458 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.636377    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 16:18:09.636458 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.636394    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:18:09.646120 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.646097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"]
Sep 11 16:18:09.672718 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.672684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.677652 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.677627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59c4f32b-34e6-479e-9005-d20271b23e28-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.677761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.677667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c4f32b-34e6-479e-9005-d20271b23e28-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.677803 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.677754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59c4f32b-34e6-479e-9005-d20271b23e28-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.677862 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.677838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwg7\" (UniqueName: \"kubernetes.io/projected/59c4f32b-34e6-479e-9005-d20271b23e28-kube-api-access-dtwg7\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.779002 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.778967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwg7\" (UniqueName: \"kubernetes.io/projected/59c4f32b-34e6-479e-9005-d20271b23e28-kube-api-access-dtwg7\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.779002 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.779007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59c4f32b-34e6-479e-9005-d20271b23e28-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.779217 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.779032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c4f32b-34e6-479e-9005-d20271b23e28-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.779217 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.779074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59c4f32b-34e6-479e-9005-d20271b23e28-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.779725 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.779702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c4f32b-34e6-479e-9005-d20271b23e28-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.781660 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.781628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59c4f32b-34e6-479e-9005-d20271b23e28-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.781729 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.781659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59c4f32b-34e6-479e-9005-d20271b23e28-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.788101 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.788082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwg7\" (UniqueName: \"kubernetes.io/projected/59c4f32b-34e6-479e-9005-d20271b23e28-kube-api-access-dtwg7\") pod \"prometheus-operator-68bcd7f987-24hv8\" (UID: \"59c4f32b-34e6-479e-9005-d20271b23e28\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:09.814361 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.814327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.940172 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:09.940131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"
Sep 11 16:18:10.209370 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.209293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-24hv8"]
Sep 11 16:18:10.211840 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:10.211801    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c4f32b_34e6_479e_9005_d20271b23e28.slice/crio-e818059fb1ec2abe4232f70504e88cf6ab32fcc952a93ba12cf2f1ea18c5f999 WatchSource:0}: Error finding container e818059fb1ec2abe4232f70504e88cf6ab32fcc952a93ba12cf2f1ea18c5f999: Status 404 returned error can't find the container with id e818059fb1ec2abe4232f70504e88cf6ab32fcc952a93ba12cf2f1ea18c5f999
Sep 11 16:18:10.265016 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.264973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8" event={"ID":"59c4f32b-34e6-479e-9005-d20271b23e28","Type":"ContainerStarted","Data":"e818059fb1ec2abe4232f70504e88cf6ab32fcc952a93ba12cf2f1ea18c5f999"}
Sep 11 16:18:10.486225 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.486137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:18:10.486225 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.486201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:18:10.489274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.489254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:18:10.490417 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.490400    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:18:10.498726 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.498703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c159bf6-5095-4cf6-b9fc-0935e2fd00ca-original-pull-secret\") pod \"global-pull-secret-syncer-gmprr\" (UID: \"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca\") " pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:18:10.498857 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.498778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cd81b90-21f6-4343-99c5-3d746e3c480e-metrics-certs\") pod \"network-metrics-daemon-lhpp4\" (UID: \"8cd81b90-21f6-4343-99c5-3d746e3c480e\") " pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:18:10.587209 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.587170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:18:10.590031 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.590013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:18:10.600798 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.600782    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:18:10.610581 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.610564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdvr\" (UniqueName: \"kubernetes.io/projected/c0eb169c-3c31-48f2-a973-53f0aacced29-kube-api-access-hgdvr\") pod \"network-check-target-bdf6j\" (UID: \"c0eb169c-3c31-48f2-a973-53f0aacced29\") " pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:18:10.749891 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.749818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gmprr"
Sep 11 16:18:10.760162 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.760138    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn9xf\""
Sep 11 16:18:10.766900 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.766883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mk9m2\""
Sep 11 16:18:10.767908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.767889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhpp4"
Sep 11 16:18:10.775224 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:10.775204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:18:11.021598 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:11.021295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gmprr"]
Sep 11 16:18:11.023727 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:11.023695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c159bf6_5095_4cf6_b9fc_0935e2fd00ca.slice/crio-f0902580b38c36d44ec7db0bdc9ba1d203d2ebc826690620577743d4a410e221 WatchSource:0}: Error finding container f0902580b38c36d44ec7db0bdc9ba1d203d2ebc826690620577743d4a410e221: Status 404 returned error can't find the container with id f0902580b38c36d44ec7db0bdc9ba1d203d2ebc826690620577743d4a410e221
Sep 11 16:18:11.047916 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:11.047890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhpp4"]
Sep 11 16:18:11.050705 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:11.050679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd81b90_21f6_4343_99c5_3d746e3c480e.slice/crio-be713df4fc9e6e94f033386bc075ab39122644f82516a2688b2d7af455729261 WatchSource:0}: Error finding container be713df4fc9e6e94f033386bc075ab39122644f82516a2688b2d7af455729261: Status 404 returned error can't find the container with id be713df4fc9e6e94f033386bc075ab39122644f82516a2688b2d7af455729261
Sep 11 16:18:11.062278 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:11.062249    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdf6j"]
Sep 11 16:18:11.065451 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:11.065428    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eb169c_3c31_48f2_a973_53f0aacced29.slice/crio-bcfd1421bd91a370c52d8adf540a17fc09cca0846e53f0d0e13d7dfed25cfb66 WatchSource:0}: Error finding container bcfd1421bd91a370c52d8adf540a17fc09cca0846e53f0d0e13d7dfed25cfb66: Status 404 returned error can't find the container with id bcfd1421bd91a370c52d8adf540a17fc09cca0846e53f0d0e13d7dfed25cfb66
Sep 11 16:18:11.268964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:11.268929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhpp4" event={"ID":"8cd81b90-21f6-4343-99c5-3d746e3c480e","Type":"ContainerStarted","Data":"be713df4fc9e6e94f033386bc075ab39122644f82516a2688b2d7af455729261"}
Sep 11 16:18:11.270107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:11.270080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdf6j" event={"ID":"c0eb169c-3c31-48f2-a973-53f0aacced29","Type":"ContainerStarted","Data":"bcfd1421bd91a370c52d8adf540a17fc09cca0846e53f0d0e13d7dfed25cfb66"}
Sep 11 16:18:11.271154 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:11.271132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gmprr" event={"ID":"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca","Type":"ContainerStarted","Data":"f0902580b38c36d44ec7db0bdc9ba1d203d2ebc826690620577743d4a410e221"}
Sep 11 16:18:12.625672 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:12.625635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:13.245043 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.245009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jzs9s"
Sep 11 16:18:13.283367 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.283323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhpp4" event={"ID":"8cd81b90-21f6-4343-99c5-3d746e3c480e","Type":"ContainerStarted","Data":"7bfc85dcc4ea90db30e0c16c9b824bf3d76b81876d286609f62d886f2bfc28c2"}
Sep 11 16:18:13.283517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.283394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhpp4" event={"ID":"8cd81b90-21f6-4343-99c5-3d746e3c480e","Type":"ContainerStarted","Data":"6959a452a9a38d9c0fecefef9860b8a96474e1a4a0a0c0a6b31acd998e400e2a"}
Sep 11 16:18:13.286042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.286012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8" event={"ID":"59c4f32b-34e6-479e-9005-d20271b23e28","Type":"ContainerStarted","Data":"18e0210df45c3e32046fd5b52b957d4e4fa0577dd728d1738218c2484897261c"}
Sep 11 16:18:13.286167 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.286066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8" event={"ID":"59c4f32b-34e6-479e-9005-d20271b23e28","Type":"ContainerStarted","Data":"6645ed41d0287bbd066ddcca129ead152dd3481bba053447f33e589726508fc6"}
Sep 11 16:18:13.300333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.299344    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lhpp4" podStartSLOduration=67.086477084 podStartE2EDuration="1m8.299325602s" totalImagesPullingTime="1.212848518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:13.298923002 +0000 UTC m=+68.980027440" watchObservedRunningTime="2026-09-11 16:18:13.299325602 +0000 UTC m=+68.980430028"
Sep 11 16:18:13.322517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.322331    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-24hv8" podStartSLOduration=2.273540492 podStartE2EDuration="4.322309575s" totalImagesPullingTime="2.048769083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:13.321281229 +0000 UTC m=+69.002385656" watchObservedRunningTime="2026-09-11 16:18:13.322309575 +0000 UTC m=+69.003414001"
Sep 11 16:18:13.422735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.422641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:13.495236 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:13.495152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:14.250624 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:14.250580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:14.323742 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:14.323695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:14.830861 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:14.830802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:14.917494 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:14.917454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.022248 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.022200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.061531 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.061494    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"]
Sep 11 16:18:15.064900 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.064875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.067750 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.067726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 16:18:15.068116 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.068100    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z49xl\""
Sep 11 16:18:15.068301 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.068283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 16:18:15.079121 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.079101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"]
Sep 11 16:18:15.095714 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.095658    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"]
Sep 11 16:18:15.099176 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.099155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.102046 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.102026    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 16:18:15.102621 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.102523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 16:18:15.102713 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.102647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jh72j\""
Sep 11 16:18:15.103421 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.103400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 16:18:15.108301 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.108279    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tzzrn"]
Sep 11 16:18:15.111387 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.111369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.114577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.114542    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:18:15.114865 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.114846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6vtlj\""
Sep 11 16:18:15.115064 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.115049    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:18:15.115243 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.115219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:18:15.115637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.115617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"]
Sep 11 16:18:15.121977 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.121958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61163c6-6f55-4aab-be6e-14b682c63639-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.122080 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.121992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fhs\" (UniqueName: \"kubernetes.io/projected/a61163c6-6f55-4aab-be6e-14b682c63639-kube-api-access-d5fhs\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.122080 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.122025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.122192 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.122116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.152235 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.152199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.222953 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.222917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-textfile\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.222953 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.222956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcmq\" (UniqueName: \"kubernetes.io/projected/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-kube-api-access-wdcmq\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.222981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67afa78c-a699-4ec5-ae58-68f01f025fa7-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.223193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fhs\" (UniqueName: \"kubernetes.io/projected/a61163c6-6f55-4aab-be6e-14b682c63639-kube-api-access-d5fhs\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.223193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfmw\" (UniqueName: \"kubernetes.io/projected/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-api-access-krfmw\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.223193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/67afa78c-a699-4ec5-ae58-68f01f025fa7-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.223365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-tls\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.223365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-metrics-client-ca\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-sys\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-root\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61163c6-6f55-4aab-be6e-14b682c63639-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-wtmp\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:18:15.223549    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.223588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.223651 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:18:15.223618    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls podName:a61163c6-6f55-4aab-be6e-14b682c63639 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:15.723598791 +0000 UTC m=+71.404703211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-vfwmr" (UID: "a61163c6-6f55-4aab-be6e-14b682c63639") : secret "openshift-state-metrics-tls" not found
Sep 11 16:18:15.224244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.224215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61163c6-6f55-4aab-be6e-14b682c63639-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.226438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.226416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.237693 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.237669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fhs\" (UniqueName: \"kubernetes.io/projected/a61163c6-6f55-4aab-be6e-14b682c63639-kube-api-access-d5fhs\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.246347 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.246311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.324274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.324274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-tls\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.324752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.324752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-metrics-client-ca\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.324752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-sys\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.324932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-root\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.324932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.324932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-wtmp\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.324932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-textfile\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcmq\" (UniqueName: \"kubernetes.io/projected/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-kube-api-access-wdcmq\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.324992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67afa78c-a699-4ec5-ae58-68f01f025fa7-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfmw\" (UniqueName: \"kubernetes.io/projected/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-api-access-krfmw\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/67afa78c-a699-4ec5-ae58-68f01f025fa7-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.325111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325441 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-sys\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325441 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-root\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325441 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-wtmp\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325596 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-textfile\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.325736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/67afa78c-a699-4ec5-ae58-68f01f025fa7-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.326004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.325971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.326091 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.326014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-metrics-client-ca\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.326179 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.326136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67afa78c-a699-4ec5-ae58-68f01f025fa7-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.327428 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.327408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-tls\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.327621 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.327595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.327928 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.327663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.328107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.328088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.340542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.340515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfmw\" (UniqueName: \"kubernetes.io/projected/67afa78c-a699-4ec5-ae58-68f01f025fa7-kube-api-access-krfmw\") pod \"kube-state-metrics-59fbdd77d-wxxcr\" (UID: \"67afa78c-a699-4ec5-ae58-68f01f025fa7\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.341038 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.341016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcmq\" (UniqueName: \"kubernetes.io/projected/4bea74c9-0f80-47a9-9625-17ca6c25cb4a-kube-api-access-wdcmq\") pod \"node-exporter-tzzrn\" (UID: \"4bea74c9-0f80-47a9-9625-17ca6c25cb4a\") " pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.352047 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.351986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.408989 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.408956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"
Sep 11 16:18:15.421357 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.421333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tzzrn"
Sep 11 16:18:15.527996 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.527961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.662058 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:15.661993    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bea74c9_0f80_47a9_9625_17ca6c25cb4a.slice/crio-19ef1789e6d53778b57c61422456aa0b465b1de7017e2c5f214d1e6d34ac194b WatchSource:0}: Error finding container 19ef1789e6d53778b57c61422456aa0b465b1de7017e2c5f214d1e6d34ac194b: Status 404 returned error can't find the container with id 19ef1789e6d53778b57c61422456aa0b465b1de7017e2c5f214d1e6d34ac194b
Sep 11 16:18:15.729814 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.729280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:15.729814 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:18:15.729533    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 16:18:15.729814 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:18:15.729602    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls podName:a61163c6-6f55-4aab-be6e-14b682c63639 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:16.729582839 +0000 UTC m=+72.410687246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-vfwmr" (UID: "a61163c6-6f55-4aab-be6e-14b682c63639") : secret "openshift-state-metrics-tls" not found
Sep 11 16:18:15.825484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.825448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.925003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.924919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.955861 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:15.955818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr"]
Sep 11 16:18:15.958772 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:15.958750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67afa78c_a699_4ec5_ae58_68f01f025fa7.slice/crio-9034de7c6c058d3c9864c5aa1845e370bf5667dad1bfef4c8a74c6f1146bc60c WatchSource:0}: Error finding container 9034de7c6c058d3c9864c5aa1845e370bf5667dad1bfef4c8a74c6f1146bc60c: Status 404 returned error can't find the container with id 9034de7c6c058d3c9864c5aa1845e370bf5667dad1bfef4c8a74c6f1146bc60c
Sep 11 16:18:16.055915 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.055881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:16.146230 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.146195    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:18:16.151503 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.151441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.154057 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.153718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:16.155669 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.155645    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 16:18:16.155669 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.155654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 16:18:16.155895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.155678    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 16:18:16.155895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.155875    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 16:18:16.155895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.155881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 16:18:16.156048 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.156028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkktx\""
Sep 11 16:18:16.156100 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.156080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 16:18:16.156155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.156101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 16:18:16.157218 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.157128    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 16:18:16.157218 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.157186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 16:18:16.180292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.180226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:18:16.233667 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.233844 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9sc\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-kube-api-access-tq9sc\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.233844 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.233844 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.233844 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-web-config\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.233980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-out\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.234005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.234040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.234037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.300304 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.300053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdf6j" event={"ID":"c0eb169c-3c31-48f2-a973-53f0aacced29","Type":"ContainerStarted","Data":"4dcb6b31cb7b4de82b1103f4957a3a7d180481229f3de79376f506c7d728fc9d"}
Sep 11 16:18:16.300304 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.300179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:18:16.302031 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.302006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gmprr" event={"ID":"1c159bf6-5095-4cf6-b9fc-0935e2fd00ca","Type":"ContainerStarted","Data":"316af8265746b7d7323bf2ef41c419aaf22a317a446f680c87fec2a1460fee60"}
Sep 11 16:18:16.303381 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.303354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzzrn" event={"ID":"4bea74c9-0f80-47a9-9625-17ca6c25cb4a","Type":"ContainerStarted","Data":"19ef1789e6d53778b57c61422456aa0b465b1de7017e2c5f214d1e6d34ac194b"}
Sep 11 16:18:16.304613 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.304588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr" event={"ID":"67afa78c-a699-4ec5-ae58-68f01f025fa7","Type":"ContainerStarted","Data":"9034de7c6c058d3c9864c5aa1845e370bf5667dad1bfef4c8a74c6f1146bc60c"}
Sep 11 16:18:16.319696 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.319655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bdf6j" podStartSLOduration=66.726052668 podStartE2EDuration="1m11.319640875s" totalImagesPullingTime="4.593588207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:16.318989958 +0000 UTC m=+72.000094384" watchObservedRunningTime="2026-09-11 16:18:16.319640875 +0000 UTC m=+72.000745336"
Sep 11 16:18:16.335102 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-web-config\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:18:16.335204    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:18:16.335265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls podName:6e80772e-9263-44ff-93b1-26a3f88f8c44 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:16.835249269 +0000 UTC m=+72.516353675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44") : secret "alertmanager-main-tls" not found
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-out\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.335597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9sc\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-kube-api-access-tq9sc\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.336443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.335769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.336569 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.336544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.337205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.337159    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gmprr" podStartSLOduration=66.687224329 podStartE2EDuration="1m11.337133807s" totalImagesPullingTime="4.649909478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:16.336709102 +0000 UTC m=+72.017813528" watchObservedRunningTime="2026-09-11 16:18:16.337133807 +0000 UTC m=+72.018238233"
Sep 11 16:18:16.337299 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.337233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.338353 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.338328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-web-config\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.339301 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.339275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.340003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.339951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.340003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.339980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.340166 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.340132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.340494 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.340471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.340886 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.340817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-out\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.341330 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.341308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.347218 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.347197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9sc\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-kube-api-access-tq9sc\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.402854 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.402803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:16.693910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.693819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:16.739605 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.739578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:16.741905 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.741886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a61163c6-6f55-4aab-be6e-14b682c63639-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vfwmr\" (UID: \"a61163c6-6f55-4aab-be6e-14b682c63639\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:16.841075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.840993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.843683 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.843658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:16.874700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:16.874656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"
Sep 11 16:18:17.065174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.065141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:17.134888 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.134850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:17.184236 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.184172    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75956f6695-ckhj7"]
Sep 11 16:18:17.187507 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.187485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.190313 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.190279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 16:18:17.190435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.190418    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-egj7gsscev381\""
Sep 11 16:18:17.191561 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.191528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 16:18:17.191661 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.191533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 16:18:17.191661 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.191568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hxwbv\""
Sep 11 16:18:17.191661 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.191634    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 16:18:17.191964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.191945    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 16:18:17.203667 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.203553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75956f6695-ckhj7"]
Sep 11 16:18:17.226869 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.226810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:17.312144 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.312030    2580 generic.go:356] "Generic (PLEG): container finished" podID="4bea74c9-0f80-47a9-9625-17ca6c25cb4a" containerID="2b130cf148ce9f36a1e75999c42b02c8cb474282e8106413f942e5ad0570a5b0" exitCode=0
Sep 11 16:18:17.312144 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.312113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzzrn" event={"ID":"4bea74c9-0f80-47a9-9625-17ca6c25cb4a","Type":"ContainerDied","Data":"2b130cf148ce9f36a1e75999c42b02c8cb474282e8106413f942e5ad0570a5b0"}
Sep 11 16:18:17.315552 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.315369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr" event={"ID":"67afa78c-a699-4ec5-ae58-68f01f025fa7","Type":"ContainerStarted","Data":"39961fc7ab310bffb780a36759bf64a772c0a66799b3ec21867a5b50c87b8ebb"}
Sep 11 16:18:17.315552 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.315433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr" event={"ID":"67afa78c-a699-4ec5-ae58-68f01f025fa7","Type":"ContainerStarted","Data":"8b388dc759680abf92374d4825d4cad1d6426ea956bf13c12f86b81c906fc50f"}
Sep 11 16:18:17.345381 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.345728 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.345728 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-grpc-tls\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.345728 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de14faed-a419-4628-a3b6-3695750ef60f-metrics-client-ca\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.345728 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.345728 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-tls\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.345728 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.346027 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.345741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kk8\" (UniqueName: \"kubernetes.io/projected/de14faed-a419-4628-a3b6-3695750ef60f-kube-api-access-v2kk8\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.347173 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.347148    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr"]
Sep 11 16:18:17.350368 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:17.350336    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61163c6_6f55_4aab_be6e_14b682c63639.slice/crio-9266f28e05493e07e70e808677feb45972b80239120a3b799e02e262bdc31ca9 WatchSource:0}: Error finding container 9266f28e05493e07e70e808677feb45972b80239120a3b799e02e262bdc31ca9: Status 404 returned error can't find the container with id 9266f28e05493e07e70e808677feb45972b80239120a3b799e02e262bdc31ca9
Sep 11 16:18:17.356383 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.355392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:18:17.366103 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:17.366073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e80772e_9263_44ff_93b1_26a3f88f8c44.slice/crio-facc2fc88b76051eb1e61aac9071cdf9a0f7da6cc3696fca7a8edabd644bf214 WatchSource:0}: Error finding container facc2fc88b76051eb1e61aac9071cdf9a0f7da6cc3696fca7a8edabd644bf214: Status 404 returned error can't find the container with id facc2fc88b76051eb1e61aac9071cdf9a0f7da6cc3696fca7a8edabd644bf214
Sep 11 16:18:17.446329 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-grpc-tls\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446401 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de14faed-a419-4628-a3b6-3695750ef60f-metrics-client-ca\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446401 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446502 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-tls\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446502 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446502 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kk8\" (UniqueName: \"kubernetes.io/projected/de14faed-a419-4628-a3b6-3695750ef60f-kube-api-access-v2kk8\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.446648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.446574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.447561 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.447222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de14faed-a419-4628-a3b6-3695750ef60f-metrics-client-ca\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.450081 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.450039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.450573 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.450200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-grpc-tls\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.450573 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.450533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.451136 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.451091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.453576 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.453431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-tls\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.453576 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.453539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de14faed-a419-4628-a3b6-3695750ef60f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.457904 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.457813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kk8\" (UniqueName: \"kubernetes.io/projected/de14faed-a419-4628-a3b6-3695750ef60f-kube-api-access-v2kk8\") pod \"thanos-querier-75956f6695-ckhj7\" (UID: \"de14faed-a419-4628-a3b6-3695750ef60f\") " pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.497484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.497394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:17.771299 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:17.771275    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75956f6695-ckhj7"]
Sep 11 16:18:17.773129 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:17.773106    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde14faed_a419_4628_a3b6_3695750ef60f.slice/crio-b05bc793d7ca545b0bee50a4b2316ebd658adf0c009d5bd81e9722ccbe10160a WatchSource:0}: Error finding container b05bc793d7ca545b0bee50a4b2316ebd658adf0c009d5bd81e9722ccbe10160a: Status 404 returned error can't find the container with id b05bc793d7ca545b0bee50a4b2316ebd658adf0c009d5bd81e9722ccbe10160a
Sep 11 16:18:18.061042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.060815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:18.159778 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.159746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:18.258278 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.258216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:18.321527 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.321428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:18.326115 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.326085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzzrn" event={"ID":"4bea74c9-0f80-47a9-9625-17ca6c25cb4a","Type":"ContainerStarted","Data":"eeea8483dc5a9b7a12ecfd35a10486803a863582c7c1a2b27e0760ddb37ef938"}
Sep 11 16:18:18.326238 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.326150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzzrn" event={"ID":"4bea74c9-0f80-47a9-9625-17ca6c25cb4a","Type":"ContainerStarted","Data":"746e931593edebd340d3ed3b42613114641c6d46a88ee3b795bdbeec36d8db1a"}
Sep 11 16:18:18.329903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.329872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr" event={"ID":"67afa78c-a699-4ec5-ae58-68f01f025fa7","Type":"ContainerStarted","Data":"24e442589c5d624a1fb589a2634e5b3d82c2c0880816bf997dfec866e516f7e3"}
Sep 11 16:18:18.332838 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.332733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"b05bc793d7ca545b0bee50a4b2316ebd658adf0c009d5bd81e9722ccbe10160a"}
Sep 11 16:18:18.335085 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.335062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr" event={"ID":"a61163c6-6f55-4aab-be6e-14b682c63639","Type":"ContainerStarted","Data":"445ee35f33d373657d0df746b885e9694ea1f646da3b50694879de2890b8439b"}
Sep 11 16:18:18.335190 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.335115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr" event={"ID":"a61163c6-6f55-4aab-be6e-14b682c63639","Type":"ContainerStarted","Data":"23877d2f5574530ab72737cd2bf6fa52c849bf899be4e88c33a8e370d6228f92"}
Sep 11 16:18:18.335190 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.335144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr" event={"ID":"a61163c6-6f55-4aab-be6e-14b682c63639","Type":"ContainerStarted","Data":"9266f28e05493e07e70e808677feb45972b80239120a3b799e02e262bdc31ca9"}
Sep 11 16:18:18.336798 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.336731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"facc2fc88b76051eb1e61aac9071cdf9a0f7da6cc3696fca7a8edabd644bf214"}
Sep 11 16:18:18.348510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.348459    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tzzrn" podStartSLOduration=2.365381896 podStartE2EDuration="3.34843929s" totalImagesPullingTime="909.32165ms" totalInitContainerRuntime="73.735744ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:18.348198203 +0000 UTC m=+74.029302629" watchObservedRunningTime="2026-09-11 16:18:18.34843929 +0000 UTC m=+74.029543716"
Sep 11 16:18:18.371796 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:18.371737    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxxcr" podStartSLOduration=2.214952502 podStartE2EDuration="3.371721892s" totalImagesPullingTime="1.15676939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:18.371478854 +0000 UTC m=+74.052583280" watchObservedRunningTime="2026-09-11 16:18:18.371721892 +0000 UTC m=+74.052826318"
Sep 11 16:18:19.342969 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.342929    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95" exitCode=0
Sep 11 16:18:19.343157 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.343027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"}
Sep 11 16:18:19.345600 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.345573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr" event={"ID":"a61163c6-6f55-4aab-be6e-14b682c63639","Type":"ContainerStarted","Data":"0c8ab1e967db825ac111a42226e31beea083c43ac9583f387cf6231dd7778182"}
Sep 11 16:18:19.394333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.394273    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vfwmr" podStartSLOduration=3.002823769 podStartE2EDuration="4.394255566s" totalImagesPullingTime="1.391431797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:19.392975606 +0000 UTC m=+75.074080032" watchObservedRunningTime="2026-09-11 16:18:19.394255566 +0000 UTC m=+75.075359993"
Sep 11 16:18:19.574984 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.574906    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d564766cb-khndb"]
Sep 11 16:18:19.579418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.579392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.582420 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.582399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 16:18:19.582575 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.582544    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eg4qoeb9de9tb\""
Sep 11 16:18:19.582790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.582759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 16:18:19.582790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.582759    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 16:18:19.582990 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.582881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xb2q8\""
Sep 11 16:18:19.583048 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.583022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 16:18:19.593071 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.593050    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d564766cb-khndb"]
Sep 11 16:18:19.663278 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa850e6a-27cc-482e-959b-81ed5b525dc1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.663363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddx5\" (UniqueName: \"kubernetes.io/projected/aa850e6a-27cc-482e-959b-81ed5b525dc1-kube-api-access-vddx5\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.663363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-secret-metrics-server-tls\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.663363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-secret-metrics-server-client-certs\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.663463 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa850e6a-27cc-482e-959b-81ed5b525dc1-audit-log\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.663463 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa850e6a-27cc-482e-959b-81ed5b525dc1-metrics-server-audit-profiles\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.663463 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.663446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-client-ca-bundle\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764271 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa850e6a-27cc-482e-959b-81ed5b525dc1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764386 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddx5\" (UniqueName: \"kubernetes.io/projected/aa850e6a-27cc-482e-959b-81ed5b525dc1-kube-api-access-vddx5\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764386 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-secret-metrics-server-tls\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764386 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-secret-metrics-server-client-certs\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa850e6a-27cc-482e-959b-81ed5b525dc1-audit-log\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa850e6a-27cc-482e-959b-81ed5b525dc1-metrics-server-audit-profiles\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-client-ca-bundle\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.764992 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.764965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa850e6a-27cc-482e-959b-81ed5b525dc1-audit-log\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.765083 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.765038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa850e6a-27cc-482e-959b-81ed5b525dc1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.765622 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.765601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa850e6a-27cc-482e-959b-81ed5b525dc1-metrics-server-audit-profiles\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.767249 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.767224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-client-ca-bundle\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.767365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.767256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-secret-metrics-server-tls\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.767365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.767349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa850e6a-27cc-482e-959b-81ed5b525dc1-secret-metrics-server-client-certs\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.774500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.774454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddx5\" (UniqueName: \"kubernetes.io/projected/aa850e6a-27cc-482e-959b-81ed5b525dc1-kube-api-access-vddx5\") pod \"metrics-server-6d564766cb-khndb\" (UID: \"aa850e6a-27cc-482e-959b-81ed5b525dc1\") " pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.818452 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.818420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:19.824601 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.824577    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"]
Sep 11 16:18:19.827700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.827683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:19.831641 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.831619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 16:18:19.831641 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.831633    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l7k48\""
Sep 11 16:18:19.840487 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.840463    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"]
Sep 11 16:18:19.888625 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.888594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:19.966356 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.966325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-ln6lc\" (UID: \"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:19.966522 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:19.966396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhg8\" (UniqueName: \"kubernetes.io/projected/38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b-kube-api-access-vdhg8\") pod \"monitoring-plugin-5d85778b9d-ln6lc\" (UID: \"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:20.067183 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.067152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhg8\" (UniqueName: \"kubernetes.io/projected/38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b-kube-api-access-vdhg8\") pod \"monitoring-plugin-5d85778b9d-ln6lc\" (UID: \"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:20.067346 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.067221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-ln6lc\" (UID: \"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:20.069976 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.069950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-ln6lc\" (UID: \"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:20.074020 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.073990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:20.076856 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.076786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhg8\" (UniqueName: \"kubernetes.io/projected/38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b-kube-api-access-vdhg8\") pod \"monitoring-plugin-5d85778b9d-ln6lc\" (UID: \"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:20.139569 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.139537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:20.161416 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.161343    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d564766cb-khndb"]
Sep 11 16:18:20.164538 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:20.164502    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa850e6a_27cc_482e_959b_81ed5b525dc1.slice/crio-92906f83ca0fc76b1bccb5225893cd0a90580984252bc2650542a88616ae4293 WatchSource:0}: Error finding container 92906f83ca0fc76b1bccb5225893cd0a90580984252bc2650542a88616ae4293: Status 404 returned error can't find the container with id 92906f83ca0fc76b1bccb5225893cd0a90580984252bc2650542a88616ae4293
Sep 11 16:18:20.228321 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.228050    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76485bf8fb-wn22k"
Sep 11 16:18:20.343756 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.343284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"]
Sep 11 16:18:20.347143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.347120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.352453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.351973    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bmz6b\""
Sep 11 16:18:20.356048 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.355932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 16:18:20.359305 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.359277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"2e392b6ff8830334956161b1da1536de2d2706a1397e9bcbdfbd2eef2ee85918"}
Sep 11 16:18:20.359400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.359386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"d1e97017556bf2bd3797e3dcb1277c182a12b2588bf7eb3e7fa250366ad44c91"}
Sep 11 16:18:20.359450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.359433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"b39e299b6461eff0b397fa079cac231059157ae33260cdbb991f2678d05eeb2d"}
Sep 11 16:18:20.361157 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.361130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d564766cb-khndb" event={"ID":"aa850e6a-27cc-482e-959b-81ed5b525dc1","Type":"ContainerStarted","Data":"92906f83ca0fc76b1bccb5225893cd0a90580984252bc2650542a88616ae4293"}
Sep 11 16:18:20.375559 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.375388    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 16:18:20.375559 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.375404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 16:18:20.375559 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.375413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 16:18:20.375954 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.375853    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 16:18:20.385019 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.384980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 16:18:20.386015 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.385991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"]
Sep 11 16:18:20.439440 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.439417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"]
Sep 11 16:18:20.449953 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.449906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:20.471555 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.471522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjsl7\" (UniqueName: \"kubernetes.io/projected/569e6a2c-6c78-4982-b64c-9dc201fe0510-kube-api-access-jjsl7\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.471711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.471568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-federate-client-tls\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.471711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.471600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.471711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.471645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-serving-certs-ca-bundle\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.472143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.471933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.472143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.471987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-secret-telemeter-client\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.472143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.472090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-telemeter-client-tls\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.472314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.472156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-metrics-client-ca\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.513650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.513619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:20.570526 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:20.570484    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c9fdc6_9d85_47fe_9f8e_4ec0527bf15b.slice/crio-d5c9d76d34ad0b78bd66cbf42c7dd35f5169570ae3992405c55734fd1572497b WatchSource:0}: Error finding container d5c9d76d34ad0b78bd66cbf42c7dd35f5169570ae3992405c55734fd1572497b: Status 404 returned error can't find the container with id d5c9d76d34ad0b78bd66cbf42c7dd35f5169570ae3992405c55734fd1572497b
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-metrics-client-ca\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjsl7\" (UniqueName: \"kubernetes.io/projected/569e6a2c-6c78-4982-b64c-9dc201fe0510-kube-api-access-jjsl7\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-federate-client-tls\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-serving-certs-ca-bundle\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-secret-telemeter-client\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.572786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.572789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-telemeter-client-tls\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.573663 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.573551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-metrics-client-ca\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.574273 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.574230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-serving-certs-ca-bundle\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.574273 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.574258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569e6a2c-6c78-4982-b64c-9dc201fe0510-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.575885 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.575861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.575986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.575866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-telemeter-client-tls\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.576166 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.576146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-federate-client-tls\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.577132 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.577113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/569e6a2c-6c78-4982-b64c-9dc201fe0510-secret-telemeter-client\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.585124 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.585103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjsl7\" (UniqueName: \"kubernetes.io/projected/569e6a2c-6c78-4982-b64c-9dc201fe0510-kube-api-access-jjsl7\") pod \"telemeter-client-56bd74bfd8-kjg7s\" (UID: \"569e6a2c-6c78-4982-b64c-9dc201fe0510\") " pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.613763 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.613713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:20.663184 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.663160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"
Sep 11 16:18:20.946937 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:20.946912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s"]
Sep 11 16:18:21.176369 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:21.176338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569e6a2c_6c78_4982_b64c_9dc201fe0510.slice/crio-eb6c66b454e4569c07898ab486ceb26e1259c36885ac6e30162ed73f71fcd261 WatchSource:0}: Error finding container eb6c66b454e4569c07898ab486ceb26e1259c36885ac6e30162ed73f71fcd261: Status 404 returned error can't find the container with id eb6c66b454e4569c07898ab486ceb26e1259c36885ac6e30162ed73f71fcd261
Sep 11 16:18:21.368125 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.367714    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:18:21.372243 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.372221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.375318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.375294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 16:18:21.375786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.375760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"55c195f0b1de1af10446d97de3e41d4569be4587a0d12ae83c68f6816f72e957"}
Sep 11 16:18:21.375960 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.375936    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 16:18:21.376076 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.375978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"18e50a17988dc9d9365444d3f9e257cc8b4f3aeebaeb2f6c8d27cb9ca1e7524a"}
Sep 11 16:18:21.376076 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.375778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9tvrh8k42v65q\""
Sep 11 16:18:21.376076 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" event={"ID":"de14faed-a419-4628-a3b6-3695750ef60f","Type":"ContainerStarted","Data":"307922bfd70f5f8774e7f5136e3c3be0623152046ccabcf4979cec7a2f555a13"}
Sep 11 16:18:21.376076 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.375904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 16:18:21.376232 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:21.376714 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2swnr\""
Sep 11 16:18:21.376814 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 16:18:21.376921 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376906    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 16:18:21.376989 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376974    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 16:18:21.377040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.376807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 16:18:21.377323 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.377166    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 16:18:21.377410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.377341    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 16:18:21.377410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.377353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 16:18:21.377410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.377362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 16:18:21.377410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.377387    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 16:18:21.377703 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.377650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s" event={"ID":"569e6a2c-6c78-4982-b64c-9dc201fe0510","Type":"ContainerStarted","Data":"eb6c66b454e4569c07898ab486ceb26e1259c36885ac6e30162ed73f71fcd261"}
Sep 11 16:18:21.379427 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.379406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc" event={"ID":"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b","Type":"ContainerStarted","Data":"d5c9d76d34ad0b78bd66cbf42c7dd35f5169570ae3992405c55734fd1572497b"}
Sep 11 16:18:21.380603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.380419    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 16:18:21.384954 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.384932    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 16:18:21.393137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.393079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:18:21.395611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.395570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:21.453894 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.453178    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7" podStartSLOduration=1.799236972 podStartE2EDuration="4.453148196s" totalImagesPullingTime="2.653911224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:21.451997261 +0000 UTC m=+77.133101687" watchObservedRunningTime="2026-09-11 16:18:21.453148196 +0000 UTC m=+77.134252623"
Sep 11 16:18:21.480232 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.480191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.480468 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.480450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481009 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.480688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm278\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-kube-api-access-bm278\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481168 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481289 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481289 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481289 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-config\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481698 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481698 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481698 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481698 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481698 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.481698 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.482098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.482098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.481799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm278\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-kube-api-access-bm278\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.582986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.582984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.583407 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.583016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.583407 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.583046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.583407 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.583254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.583861 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.583772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.584433 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.584087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.584627 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.584604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.585314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.585288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.585401 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.583047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.585401 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.585361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.585521 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.585419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-config\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.585521 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.585436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.586361 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.586336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.586704 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.586679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.591966 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.591142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.591966 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.591481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.591966 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.591505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.591966 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.591787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.591966 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.591911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-config\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.593042 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.592999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.593517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.593494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.593700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.593680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.594131 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.594112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.594517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.594481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.595229 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.595186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.595506 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.595487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm278\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-kube-api-access-bm278\") pod \"prometheus-k8s-0\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:21.688614 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:21.688570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:22.028969 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.028938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:18:22.044234 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:22.044198    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887cb074_4c71_4a3c_82d1_d5c628443f1d.slice/crio-d183e99e42c3f6318f3130e5fde71388764612676e0cc5b37cc44da8ee713c78 WatchSource:0}: Error finding container d183e99e42c3f6318f3130e5fde71388764612676e0cc5b37cc44da8ee713c78: Status 404 returned error can't find the container with id d183e99e42c3f6318f3130e5fde71388764612676e0cc5b37cc44da8ee713c78
Sep 11 16:18:22.326856 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.326701    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-r7ffb"]
Sep 11 16:18:22.330421 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.330397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:22.333201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.333178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-v42wt\""
Sep 11 16:18:22.333321 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.333261    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 16:18:22.333435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.333419    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 16:18:22.340329 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.340305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-r7ffb"]
Sep 11 16:18:22.341574 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.341548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:22.388262 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.388221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"}
Sep 11 16:18:22.388418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.388278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"}
Sep 11 16:18:22.388418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.388306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"}
Sep 11 16:18:22.388418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.388337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"}
Sep 11 16:18:22.388418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.388380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"}
Sep 11 16:18:22.388418 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.388409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerStarted","Data":"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"}
Sep 11 16:18:22.389777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.389752    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27" exitCode=0
Sep 11 16:18:22.389909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.389843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27"}
Sep 11 16:18:22.389970 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.389911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"d183e99e42c3f6318f3130e5fde71388764612676e0cc5b37cc44da8ee713c78"}
Sep 11 16:18:22.391533 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.391464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d564766cb-khndb" event={"ID":"aa850e6a-27cc-482e-959b-81ed5b525dc1","Type":"ContainerStarted","Data":"a3b01864a5a951b90badca9b7346fe325fbca5caa64ddc192918367c42ac687b"}
Sep 11 16:18:22.419446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.419394    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.016160721 podStartE2EDuration="6.419376193s" totalImagesPullingTime="3.373237583s" totalInitContainerRuntime="29.977889ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:22.417581954 +0000 UTC m=+78.098686423" watchObservedRunningTime="2026-09-11 16:18:22.419376193 +0000 UTC m=+78.100480617"
Sep 11 16:18:22.448480 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.448434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:22.496804 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.495532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgsc\" (UniqueName: \"kubernetes.io/projected/26439149-ee7a-46ec-8eaf-8544b10dab09-kube-api-access-2rgsc\") pod \"downloads-76f99584f4-r7ffb\" (UID: \"26439149-ee7a-46ec-8eaf-8544b10dab09\") " pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:22.542850 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.542755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:22.598552 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.598503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgsc\" (UniqueName: \"kubernetes.io/projected/26439149-ee7a-46ec-8eaf-8544b10dab09-kube-api-access-2rgsc\") pod \"downloads-76f99584f4-r7ffb\" (UID: \"26439149-ee7a-46ec-8eaf-8544b10dab09\") " pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:22.610062 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.610030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgsc\" (UniqueName: \"kubernetes.io/projected/26439149-ee7a-46ec-8eaf-8544b10dab09-kube-api-access-2rgsc\") pod \"downloads-76f99584f4-r7ffb\" (UID: \"26439149-ee7a-46ec-8eaf-8544b10dab09\") " pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:22.642262 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.642228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:22.643443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.643393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:22.818538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:22.818456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.012585 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:23.012538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.073871 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:23.073775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.208014 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:23.207933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.694112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:23.692255    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d564766cb-khndb" podStartSLOduration=3.042732491 podStartE2EDuration="4.692228946s" totalImagesPullingTime="1.649496455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:22.475249077 +0000 UTC m=+78.156353503" watchObservedRunningTime="2026-09-11 16:18:23.692228946 +0000 UTC m=+79.373333371"
Sep 11 16:18:23.694528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:23.694459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-r7ffb"]
Sep 11 16:18:23.699298 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:23.699270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26439149_ee7a_46ec_8eaf_8544b10dab09.slice/crio-1fe5535050b46e5aee9e2edd4159f2969f7b3751f1ac7c9c2b765604bbb5bbbb WatchSource:0}: Error finding container 1fe5535050b46e5aee9e2edd4159f2969f7b3751f1ac7c9c2b765604bbb5bbbb: Status 404 returned error can't find the container with id 1fe5535050b46e5aee9e2edd4159f2969f7b3751f1ac7c9c2b765604bbb5bbbb
Sep 11 16:18:24.303967 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.303934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:24.404052 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.404012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:24.408473 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.408442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc" event={"ID":"38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b","Type":"ContainerStarted","Data":"dd0c5f7fac900e7f6bf57227987b1e53c5b3ad7487e22011d5cb44fa7b5d2ebf"}
Sep 11 16:18:24.408924 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.408877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:24.412374 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.412348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s" event={"ID":"569e6a2c-6c78-4982-b64c-9dc201fe0510","Type":"ContainerStarted","Data":"d0ff0869af7dcc60e2a3f5b506397a7734445d6a4b6dffacf3e1d531f58cc956"}
Sep 11 16:18:24.412470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.412401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s" event={"ID":"569e6a2c-6c78-4982-b64c-9dc201fe0510","Type":"ContainerStarted","Data":"c129864bd480d9d3077638e52c0048120f4f0689e89aa3aea76add017c6318af"}
Sep 11 16:18:24.412470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.412428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s" event={"ID":"569e6a2c-6c78-4982-b64c-9dc201fe0510","Type":"ContainerStarted","Data":"fe48d5a7a2f722fa616236c430ec9509e9bfe45f8e57fc2978335e270bd39840"}
Sep 11 16:18:24.414768 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.414717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-r7ffb" event={"ID":"26439149-ee7a-46ec-8eaf-8544b10dab09","Type":"ContainerStarted","Data":"1fe5535050b46e5aee9e2edd4159f2969f7b3751f1ac7c9c2b765604bbb5bbbb"}
Sep 11 16:18:24.415526 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.415503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc"
Sep 11 16:18:24.448858 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.448442    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ln6lc" podStartSLOduration=2.615105962 podStartE2EDuration="5.448420216s" totalImagesPullingTime="2.833314254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:24.430188432 +0000 UTC m=+80.111292872" watchObservedRunningTime="2026-09-11 16:18:24.448420216 +0000 UTC m=+80.129524640"
Sep 11 16:18:24.473956 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.473894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56bd74bfd8-kjg7s" podStartSLOduration=2.242708749 podStartE2EDuration="4.473870225s" totalImagesPullingTime="2.231161476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:24.470723291 +0000 UTC m=+80.151827717" watchObservedRunningTime="2026-09-11 16:18:24.473870225 +0000 UTC m=+80.154974655"
Sep 11 16:18:24.479575 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:24.479547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:25.038998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:25.038956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:25.098464 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:25.098423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:26.155439 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:26.155383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:26.281882 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:26.281845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:27.401133 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:27.401109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75956f6695-ckhj7"
Sep 11 16:18:27.438212 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:27.438170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b"}
Sep 11 16:18:27.438320 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:27.438282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78"}
Sep 11 16:18:27.438377 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:27.438320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8"}
Sep 11 16:18:27.438377 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:27.438359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22"}
Sep 11 16:18:28.447864 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:28.447803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1"}
Sep 11 16:18:28.448287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:28.447915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerStarted","Data":"bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8"}
Sep 11 16:18:28.493291 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:28.493223    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.782252817 podStartE2EDuration="7.493200231s" totalImagesPullingTime="4.687900962s" totalInitContainerRuntime="23.046452ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:28.488282635 +0000 UTC m=+84.169387062" watchObservedRunningTime="2026-09-11 16:18:28.493200231 +0000 UTC m=+84.174304653"
Sep 11 16:18:28.964415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:28.964373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:29.065050 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:29.065017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:29.165006 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:29.164954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:29.270174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:29.270085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:29.341473 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:29.341437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:30.230015 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:30.229962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:30.328497 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:30.328458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:31.688940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:31.688906    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:32.238467 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.238428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.381425 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.381391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.597995 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.597917    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7688974f-n49fz"]
Sep 11 16:18:32.603488 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.603462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.607967 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.607848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 16:18:32.607967 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.607884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8xxfj\""
Sep 11 16:18:32.608137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.607985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 16:18:32.608137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.607992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 16:18:32.608137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.608069    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 16:18:32.608137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.608128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 16:18:32.615705 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.615686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7688974f-n49fz"]
Sep 11 16:18:32.699470 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.699424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-oauth-config\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.699927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.699476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z82l\" (UniqueName: \"kubernetes.io/projected/cceaeea7-9327-489b-8633-b6187c2fe943-kube-api-access-8z82l\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.699927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.699564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-oauth-serving-cert\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.699927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.699672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-serving-cert\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.699927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.699708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-service-ca\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.699927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.699749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-console-config\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.800806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.800762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-serving-cert\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.800983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.800816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-service-ca\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.800983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.800879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-console-config\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.800983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.800939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-oauth-config\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.800983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.800965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z82l\" (UniqueName: \"kubernetes.io/projected/cceaeea7-9327-489b-8633-b6187c2fe943-kube-api-access-8z82l\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.801188 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.801000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-oauth-serving-cert\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.801762 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.801678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-service-ca\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.801762 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.801693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-oauth-serving-cert\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.801976 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.801788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-console-config\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.803613 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.803588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-serving-cert\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.803691 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.803658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-oauth-config\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.809684 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.809663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z82l\" (UniqueName: \"kubernetes.io/projected/cceaeea7-9327-489b-8633-b6187c2fe943-kube-api-access-8z82l\") pod \"console-7b7688974f-n49fz\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") " pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:32.916339 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:32.916297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:33.192901 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:33.192795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:33.204364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:33.204332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7688974f-n49fz"]
Sep 11 16:18:33.207900 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:33.207867    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceaeea7_9327_489b_8633_b6187c2fe943.slice/crio-0b5a3858285700bda6a38208bb8d6ed0fdf56de189752667913377e7862b30b8 WatchSource:0}: Error finding container 0b5a3858285700bda6a38208bb8d6ed0fdf56de189752667913377e7862b30b8: Status 404 returned error can't find the container with id 0b5a3858285700bda6a38208bb8d6ed0fdf56de189752667913377e7862b30b8
Sep 11 16:18:33.475419 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:33.475334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7688974f-n49fz" event={"ID":"cceaeea7-9327-489b-8633-b6187c2fe943","Type":"ContainerStarted","Data":"0b5a3858285700bda6a38208bb8d6ed0fdf56de189752667913377e7862b30b8"}
Sep 11 16:18:36.295142 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:36.295001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:36.412225 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:36.412149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:36.488439 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:36.488405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:37.152639 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:37.152604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:37.258699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:37.258659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:38.133365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:38.133323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:38.259893 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:38.259856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:39.889240 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:39.889198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:39.889637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:39.889268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:40.272458 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.272365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:40.354859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.354805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:40.480704 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.480669    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5789865dc8-mrxb2"]
Sep 11 16:18:40.513502 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.513472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.525435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.525379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5789865dc8-mrxb2"]
Sep 11 16:18:40.553816 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.553787    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 16:18:40.579266 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-service-ca\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.579410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvcc\" (UniqueName: \"kubernetes.io/projected/05c59b15-448a-420d-84ed-f06405931ed7-kube-api-access-dzvcc\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.579410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-serving-cert\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.579410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-trusted-ca-bundle\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.579550 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-oauth-serving-cert\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.579550 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-oauth-config\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.579550 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.579503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-console-config\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.680663 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-oauth-config\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.680855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-console-config\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.680855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-service-ca\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.680855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvcc\" (UniqueName: \"kubernetes.io/projected/05c59b15-448a-420d-84ed-f06405931ed7-kube-api-access-dzvcc\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-serving-cert\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-trusted-ca-bundle\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.680979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-oauth-serving-cert\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681798 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.681721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-oauth-serving-cert\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681798 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.681731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-console-config\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.681868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-service-ca\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.681998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.681974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-trusted-ca-bundle\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.683521 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.683490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-oauth-config\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.683883 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.683862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-serving-cert\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.692326 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.692307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvcc\" (UniqueName: \"kubernetes.io/projected/05c59b15-448a-420d-84ed-f06405931ed7-kube-api-access-dzvcc\") pod \"console-5789865dc8-mrxb2\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") " pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:40.825864 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:40.825774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:41.958111 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:41.957997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:42.109040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:42.109003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:43.037990 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.037966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5789865dc8-mrxb2"]
Sep 11 16:18:43.039730 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:18:43.039708    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c59b15_448a_420d_84ed_f06405931ed7.slice/crio-5c4b9f983617a7903a9b11220d9c840fb07aee5c3b1fc06fc5a730514dabef5e WatchSource:0}: Error finding container 5c4b9f983617a7903a9b11220d9c840fb07aee5c3b1fc06fc5a730514dabef5e: Status 404 returned error can't find the container with id 5c4b9f983617a7903a9b11220d9c840fb07aee5c3b1fc06fc5a730514dabef5e
Sep 11 16:18:43.531363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.531320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-r7ffb" event={"ID":"26439149-ee7a-46ec-8eaf-8544b10dab09","Type":"ContainerStarted","Data":"9af9fa63abf79fc03d1abd2a43f6fa4249ccf3cc53602310ce64bf8671099aa8"}
Sep 11 16:18:43.531541 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.531393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:43.533785 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.533754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5789865dc8-mrxb2" event={"ID":"05c59b15-448a-420d-84ed-f06405931ed7","Type":"ContainerStarted","Data":"1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf"}
Sep 11 16:18:43.533934 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.533802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5789865dc8-mrxb2" event={"ID":"05c59b15-448a-420d-84ed-f06405931ed7","Type":"ContainerStarted","Data":"5c4b9f983617a7903a9b11220d9c840fb07aee5c3b1fc06fc5a730514dabef5e"}
Sep 11 16:18:43.535841 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.535800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7688974f-n49fz" event={"ID":"cceaeea7-9327-489b-8633-b6187c2fe943","Type":"ContainerStarted","Data":"b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232"}
Sep 11 16:18:43.542328 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.542308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-r7ffb"
Sep 11 16:18:43.554569 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.554518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-r7ffb" podStartSLOduration=2.485279431 podStartE2EDuration="21.554499221s" totalImagesPullingTime="19.06921979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:43.55143967 +0000 UTC m=+99.232544098" watchObservedRunningTime="2026-09-11 16:18:43.554499221 +0000 UTC m=+99.235603661"
Sep 11 16:18:43.614179 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.614127    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7688974f-n49fz" podStartSLOduration=2.065685163 podStartE2EDuration="11.614109144s" totalImagesPullingTime="9.548423981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:43.608127955 +0000 UTC m=+99.289232394" watchObservedRunningTime="2026-09-11 16:18:43.614109144 +0000 UTC m=+99.295213573"
Sep 11 16:18:43.636775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:43.636715    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5789865dc8-mrxb2" podStartSLOduration=3.636695157 podStartE2EDuration="3.636695157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:43.635785101 +0000 UTC m=+99.316889527" watchObservedRunningTime="2026-09-11 16:18:43.636695157 +0000 UTC m=+99.317799582"
Sep 11 16:18:44.294982 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:44.294940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.410544 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:44.410509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.506771 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:44.506735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:45.443057 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:45.443014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:45.500591 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:45.500559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.361446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:46.361403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.461768 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:46.461734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.318635 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:47.318600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bdf6j"
Sep 11 16:18:47.938009 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:47.937976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.039632 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:48.039599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.203569 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:49.203537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.304104 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:49.304061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:50.826932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:50.826903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:50.827397 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:50.826987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:50.832238 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:50.832218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:51.580553 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:51.580517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:18:51.640015 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:51.639978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7688974f-n49fz"]
Sep 11 16:18:51.771323 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:51.771284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:51.837085 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:51.837002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:52.280178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:52.280145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:52.916956 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:52.916917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:18:54.638113 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:54.638067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:54.715650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:54.715615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:59.268609 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:59.268571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:59.366841 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:59.366794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:59.895017 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:59.894990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:18:59.898661 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:18:59.898639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d564766cb-khndb"
Sep 11 16:19:04.427900 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:04.427867    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:04.491746 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:04.491711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:08.302617 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:08.302575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:08.403248 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:08.403217    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:08.721003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:08.720966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:08.788362 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:08.788330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:16.666550 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:16.666510    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b7688974f-n49fz" podUID="cceaeea7-9327-489b-8633-b6187c2fe943" containerName="console" containerID="cri-o://b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232" gracePeriod=15
Sep 11 16:19:16.950946 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:16.950919    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:19:17.015492 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-serving-cert\") pod \"cceaeea7-9327-489b-8633-b6187c2fe943\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") "
Sep 11 16:19:17.015492 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-service-ca\") pod \"cceaeea7-9327-489b-8633-b6187c2fe943\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") "
Sep 11 16:19:17.015715 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-console-config\") pod \"cceaeea7-9327-489b-8633-b6187c2fe943\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") "
Sep 11 16:19:17.015715 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-oauth-config\") pod \"cceaeea7-9327-489b-8633-b6187c2fe943\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") "
Sep 11 16:19:17.015715 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z82l\" (UniqueName: \"kubernetes.io/projected/cceaeea7-9327-489b-8633-b6187c2fe943-kube-api-access-8z82l\") pod \"cceaeea7-9327-489b-8633-b6187c2fe943\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") "
Sep 11 16:19:17.015897 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-console-config" (OuterVolumeSpecName: "console-config") pod "cceaeea7-9327-489b-8633-b6187c2fe943" (UID: "cceaeea7-9327-489b-8633-b6187c2fe943"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:17.015951 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.015926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-service-ca" (OuterVolumeSpecName: "service-ca") pod "cceaeea7-9327-489b-8633-b6187c2fe943" (UID: "cceaeea7-9327-489b-8633-b6187c2fe943"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:17.016026 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.016000    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-console-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:17.016026 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.016028    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-service-ca\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:17.017779 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.017750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cceaeea7-9327-489b-8633-b6187c2fe943" (UID: "cceaeea7-9327-489b-8633-b6187c2fe943"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:17.017892 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.017778    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cceaeea7-9327-489b-8633-b6187c2fe943-kube-api-access-8z82l" (OuterVolumeSpecName: "kube-api-access-8z82l") pod "cceaeea7-9327-489b-8633-b6187c2fe943" (UID: "cceaeea7-9327-489b-8633-b6187c2fe943"). InnerVolumeSpecName "kube-api-access-8z82l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:17.017892 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.017806    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cceaeea7-9327-489b-8633-b6187c2fe943" (UID: "cceaeea7-9327-489b-8633-b6187c2fe943"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:17.116425 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.116390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-oauth-serving-cert\") pod \"cceaeea7-9327-489b-8633-b6187c2fe943\" (UID: \"cceaeea7-9327-489b-8633-b6187c2fe943\") "
Sep 11 16:19:17.116692 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.116676    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-serving-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:17.116775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.116698    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cceaeea7-9327-489b-8633-b6187c2fe943-console-oauth-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:17.116775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.116717    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z82l\" (UniqueName: \"kubernetes.io/projected/cceaeea7-9327-489b-8633-b6187c2fe943-kube-api-access-8z82l\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:17.116775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.116752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cceaeea7-9327-489b-8633-b6187c2fe943" (UID: "cceaeea7-9327-489b-8633-b6187c2fe943"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:17.218048 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.217978    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cceaeea7-9327-489b-8633-b6187c2fe943-oauth-serving-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:17.466071 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.466035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:17.554910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.554806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:17.651238 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.651204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:17.688793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.688767    2580 generic.go:356] "Generic (PLEG): container finished" podID="cceaeea7-9327-489b-8633-b6187c2fe943" containerID="b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232" exitCode=0
Sep 11 16:19:17.689196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.688847    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7688974f-n49fz"
Sep 11 16:19:17.689196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.688855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7688974f-n49fz" event={"ID":"cceaeea7-9327-489b-8633-b6187c2fe943","Type":"ContainerDied","Data":"b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232"}
Sep 11 16:19:17.689196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.688946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7688974f-n49fz" event={"ID":"cceaeea7-9327-489b-8633-b6187c2fe943","Type":"ContainerDied","Data":"0b5a3858285700bda6a38208bb8d6ed0fdf56de189752667913377e7862b30b8"}
Sep 11 16:19:17.699786 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.699768    2580 scope.go:117] "RemoveContainer" containerID="b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232"
Sep 11 16:19:17.711973 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.711951    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7688974f-n49fz"]
Sep 11 16:19:17.713221 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.713203    2580 scope.go:117] "RemoveContainer" containerID="b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232"
Sep 11 16:19:17.713499 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:17.713479    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232\": container with ID starting with b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232 not found: ID does not exist" containerID="b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232"
Sep 11 16:19:17.713563 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.713506    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232"} err="failed to get container status \"b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232\": rpc error: code = NotFound desc = could not find container \"b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232\": container with ID starting with b60ea7091f62bf8a52c9894564741bc595cd6a12c894b82238465b43a0bbe232 not found: ID does not exist"
Sep 11 16:19:17.718431 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:17.718408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b7688974f-n49fz"]
Sep 11 16:19:18.940389 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:18.940359    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cceaeea7-9327-489b-8633-b6187c2fe943" path="/var/lib/kubelet/pods/cceaeea7-9327-489b-8633-b6187c2fe943/volumes"
Sep 11 16:19:21.689958 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:21.689920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:21.748936 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:21.748907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:21.765293 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:21.765260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:31.245549 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:31.245514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:31.322276 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:31.322244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:35.506525 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506490    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:35.506955 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506749    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="alertmanager" containerID="cri-o://876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40" gracePeriod=120
Sep 11 16:19:35.506955 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-metric" containerID="cri-o://bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd" gracePeriod=120
Sep 11 16:19:35.506955 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506883    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="prom-label-proxy" containerID="cri-o://bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc" gracePeriod=120
Sep 11 16:19:35.507139 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506926    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-web" containerID="cri-o://f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1" gracePeriod=120
Sep 11 16:19:35.507139 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy" containerID="cri-o://ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43" gracePeriod=120
Sep 11 16:19:35.507139 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.506926    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="config-reloader" containerID="cri-o://8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2" gracePeriod=120
Sep 11 16:19:35.773765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773687    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc" exitCode=0
Sep 11 16:19:35.773765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773712    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43" exitCode=0
Sep 11 16:19:35.773765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773720    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2" exitCode=0
Sep 11 16:19:35.773765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773726    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40" exitCode=0
Sep 11 16:19:35.773765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"}
Sep 11 16:19:35.774027 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"}
Sep 11 16:19:35.774027 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"}
Sep 11 16:19:35.774027 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:35.773863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"}
Sep 11 16:19:36.765530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.765507    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:36.781073 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781039    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd" exitCode=0
Sep 11 16:19:36.781073 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781071    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerID="f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1" exitCode=0
Sep 11 16:19:36.781222 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781115    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:36.781222 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"}
Sep 11 16:19:36.781222 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"}
Sep 11 16:19:36.781318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e80772e-9263-44ff-93b1-26a3f88f8c44","Type":"ContainerDied","Data":"facc2fc88b76051eb1e61aac9071cdf9a0f7da6cc3696fca7a8edabd644bf214"}
Sep 11 16:19:36.781318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.781274    2580 scope.go:117] "RemoveContainer" containerID="bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"
Sep 11 16:19:36.788281 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.788265    2580 scope.go:117] "RemoveContainer" containerID="bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"
Sep 11 16:19:36.795067 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.795040    2580 scope.go:117] "RemoveContainer" containerID="ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"
Sep 11 16:19:36.801701 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.801684    2580 scope.go:117] "RemoveContainer" containerID="f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"
Sep 11 16:19:36.815450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.813727    2580 scope.go:117] "RemoveContainer" containerID="8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"
Sep 11 16:19:36.826194 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.826169    2580 scope.go:117] "RemoveContainer" containerID="876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"
Sep 11 16:19:36.835834 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.835804    2580 scope.go:117] "RemoveContainer" containerID="e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"
Sep 11 16:19:36.842621 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.842601    2580 scope.go:117] "RemoveContainer" containerID="bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"
Sep 11 16:19:36.842913 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.842893    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc\": container with ID starting with bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc not found: ID does not exist" containerID="bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"
Sep 11 16:19:36.843008 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.842926    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"} err="failed to get container status \"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc\": rpc error: code = NotFound desc = could not find container \"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc\": container with ID starting with bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc not found: ID does not exist"
Sep 11 16:19:36.843008 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.842951    2580 scope.go:117] "RemoveContainer" containerID="bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"
Sep 11 16:19:36.843224 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.843204    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd\": container with ID starting with bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd not found: ID does not exist" containerID="bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"
Sep 11 16:19:36.843293 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.843234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"} err="failed to get container status \"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd\": rpc error: code = NotFound desc = could not find container \"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd\": container with ID starting with bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd not found: ID does not exist"
Sep 11 16:19:36.843293 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.843256    2580 scope.go:117] "RemoveContainer" containerID="ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"
Sep 11 16:19:36.843530 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.843511    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43\": container with ID starting with ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43 not found: ID does not exist" containerID="ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"
Sep 11 16:19:36.843579 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.843534    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"} err="failed to get container status \"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43\": rpc error: code = NotFound desc = could not find container \"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43\": container with ID starting with ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43 not found: ID does not exist"
Sep 11 16:19:36.843579 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.843548    2580 scope.go:117] "RemoveContainer" containerID="f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"
Sep 11 16:19:36.843847 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.843817    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1\": container with ID starting with f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1 not found: ID does not exist" containerID="f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"
Sep 11 16:19:36.843902 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.843852    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"} err="failed to get container status \"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1\": rpc error: code = NotFound desc = could not find container \"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1\": container with ID starting with f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1 not found: ID does not exist"
Sep 11 16:19:36.843902 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.843882    2580 scope.go:117] "RemoveContainer" containerID="8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"
Sep 11 16:19:36.844113 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.844097    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2\": container with ID starting with 8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2 not found: ID does not exist" containerID="8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"
Sep 11 16:19:36.844160 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844116    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"} err="failed to get container status \"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2\": rpc error: code = NotFound desc = could not find container \"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2\": container with ID starting with 8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2 not found: ID does not exist"
Sep 11 16:19:36.844160 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844129    2580 scope.go:117] "RemoveContainer" containerID="876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"
Sep 11 16:19:36.844365 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.844349    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40\": container with ID starting with 876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40 not found: ID does not exist" containerID="876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"
Sep 11 16:19:36.844408 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844370    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"} err="failed to get container status \"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40\": rpc error: code = NotFound desc = could not find container \"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40\": container with ID starting with 876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40 not found: ID does not exist"
Sep 11 16:19:36.844408 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844384    2580 scope.go:117] "RemoveContainer" containerID="e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"
Sep 11 16:19:36.844619 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:36.844606    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95\": container with ID starting with e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95 not found: ID does not exist" containerID="e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"
Sep 11 16:19:36.844668 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844620    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"} err="failed to get container status \"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95\": rpc error: code = NotFound desc = could not find container \"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95\": container with ID starting with e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95 not found: ID does not exist"
Sep 11 16:19:36.844668 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844631    2580 scope.go:117] "RemoveContainer" containerID="bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"
Sep 11 16:19:36.844891 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844865    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc"} err="failed to get container status \"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc\": rpc error: code = NotFound desc = could not find container \"bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc\": container with ID starting with bc1501ebee1a3a301fe6ab10ed28df19ed6ea3a24ac481fb6d98397c8bb22fdc not found: ID does not exist"
Sep 11 16:19:36.844960 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.844895    2580 scope.go:117] "RemoveContainer" containerID="bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"
Sep 11 16:19:36.845161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.845144    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd"} err="failed to get container status \"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd\": rpc error: code = NotFound desc = could not find container \"bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd\": container with ID starting with bc361509c82e41aa45cee8c76f37e492f783345969fa9f29124c9ec958895dfd not found: ID does not exist"
Sep 11 16:19:36.845216 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.845162    2580 scope.go:117] "RemoveContainer" containerID="ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"
Sep 11 16:19:36.845402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.845387    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43"} err="failed to get container status \"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43\": rpc error: code = NotFound desc = could not find container \"ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43\": container with ID starting with ae4301b627b8a757ce9a54fef8339710bcb7356cf125b5abf3b44ef7b379dc43 not found: ID does not exist"
Sep 11 16:19:36.845402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.845401    2580 scope.go:117] "RemoveContainer" containerID="f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"
Sep 11 16:19:36.845645 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.845628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1"} err="failed to get container status \"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1\": rpc error: code = NotFound desc = could not find container \"f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1\": container with ID starting with f515277c8b83f1d332ef8647539e768c193522e40cebc17627d7397c545f01a1 not found: ID does not exist"
Sep 11 16:19:36.845712 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.845646    2580 scope.go:117] "RemoveContainer" containerID="8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"
Sep 11 16:19:36.846026 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.846003    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2"} err="failed to get container status \"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2\": rpc error: code = NotFound desc = could not find container \"8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2\": container with ID starting with 8e95165389844cf32a3ffe4e8c8f6d882d203dd20ba0172e8bf520f93311dde2 not found: ID does not exist"
Sep 11 16:19:36.846165 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.846027    2580 scope.go:117] "RemoveContainer" containerID="876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"
Sep 11 16:19:36.846391 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.846370    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40"} err="failed to get container status \"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40\": rpc error: code = NotFound desc = could not find container \"876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40\": container with ID starting with 876ec6396173aec148009452e0be798fa9e1a8f4531bf54de5de494a255fac40 not found: ID does not exist"
Sep 11 16:19:36.846486 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.846391    2580 scope.go:117] "RemoveContainer" containerID="e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"
Sep 11 16:19:36.846705 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.846687    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95"} err="failed to get container status \"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95\": rpc error: code = NotFound desc = could not find container \"e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95\": container with ID starting with e9bb76bb209204ef7ea9b91cf84c8a0f2a4fce89a4c837fdead20ead9be25b95 not found: ID does not exist"
Sep 11 16:19:36.895900 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.895872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-out\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.895932    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-web-config\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.895988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq9sc\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-kube-api-access-tq9sc\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-volume\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-trusted-ca-bundle\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-metrics-client-ca\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-cluster-tls-config\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-tls-assets\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.896453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-main-db\") pod \"6e80772e-9263-44ff-93b1-26a3f88f8c44\" (UID: \"6e80772e-9263-44ff-93b1-26a3f88f8c44\") "
Sep 11 16:19:36.897365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:36.897365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.896971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:36.897365 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.897088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:36.899253 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.899225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-volume" (OuterVolumeSpecName: "config-volume") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.899382 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.899344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.899617 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.899592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-out" (OuterVolumeSpecName: "config-out") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:36.899700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.899659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.899700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.899669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-kube-api-access-tq9sc" (OuterVolumeSpecName: "kube-api-access-tq9sc") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "kube-api-access-tq9sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:36.899854 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.899835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:36.900114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.900099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.900244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.900226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.903468 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.903446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.910201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.910181    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-web-config" (OuterVolumeSpecName: "web-config") pod "6e80772e-9263-44ff-93b1-26a3f88f8c44" (UID: "6e80772e-9263-44ff-93b1-26a3f88f8c44"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:36.997799 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997765    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-out\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.997799 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997798    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-web-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.997799 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997811    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9sc\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-kube-api-access-tq9sc\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997849    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-config-volume\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997863    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997876    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997889    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997903    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e80772e-9263-44ff-93b1-26a3f88f8c44-metrics-client-ca\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997916    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-main-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997929    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-cluster-tls-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997942    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e80772e-9263-44ff-93b1-26a3f88f8c44-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997955    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e80772e-9263-44ff-93b1-26a3f88f8c44-tls-assets\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:36.998114 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:36.997968    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e80772e-9263-44ff-93b1-26a3f88f8c44-alertmanager-main-db\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:37.100432 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:37.100344    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:37.108677 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:37.108645    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:38.430227 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430190    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79d4f95b6f-4vfrw"]
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430517    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cceaeea7-9327-489b-8633-b6187c2fe943" containerName="console"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430531    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cceaeea7-9327-489b-8633-b6187c2fe943" containerName="console"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="init-config-reloader"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430552    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="init-config-reloader"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430561    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-web"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430567    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-web"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430574    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="config-reloader"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430579    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="config-reloader"
Sep 11 16:19:38.430587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430588    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="prom-label-proxy"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430595    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="prom-label-proxy"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="alertmanager"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430616    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="alertmanager"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430624    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430629    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430637    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-metric"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430642    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-metric"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430697    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-web"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="config-reloader"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430715    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="alertmanager"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430720    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy-metric"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430726    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cceaeea7-9327-489b-8633-b6187c2fe943" containerName="console"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430731    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="kube-rbac-proxy"
Sep 11 16:19:38.430866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.430736    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" containerName="prom-label-proxy"
Sep 11 16:19:38.433785 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.433763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.448327 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.448305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d4f95b6f-4vfrw"]
Sep 11 16:19:38.495598 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.495560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:38.510836 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.510800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-trusted-ca-bundle\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.510941 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.510852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-oauth-serving-cert\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.510941 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.510912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-serving-cert\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.511013 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.510978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-service-ca\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.511013 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.511005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-oauth-config\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.511082 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.511028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnkzn\" (UniqueName: \"kubernetes.io/projected/94189701-8e38-4499-855a-581b0e81a46c-kube-api-access-tnkzn\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.511082 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.511050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-console-config\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612416 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-service-ca\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-oauth-config\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnkzn\" (UniqueName: \"kubernetes.io/projected/94189701-8e38-4499-855a-581b0e81a46c-kube-api-access-tnkzn\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-console-config\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-trusted-ca-bundle\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-oauth-serving-cert\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.612787 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.612564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-serving-cert\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.613256 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.613227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-console-config\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.613256 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.613243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-service-ca\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.613384 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.613251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-oauth-serving-cert\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.613482 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.613458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-trusted-ca-bundle\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.614935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.614910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-serving-cert\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.615023 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.614956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-oauth-config\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.621504 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.621481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnkzn\" (UniqueName: \"kubernetes.io/projected/94189701-8e38-4499-855a-581b0e81a46c-kube-api-access-tnkzn\") pod \"console-79d4f95b6f-4vfrw\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") " pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.743185 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.743099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:38.939750 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:38.939719    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e80772e-9263-44ff-93b1-26a3f88f8c44" path="/var/lib/kubelet/pods/6e80772e-9263-44ff-93b1-26a3f88f8c44/volumes"
Sep 11 16:19:39.014749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.014675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d4f95b6f-4vfrw"]
Sep 11 16:19:39.018214 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:19:39.018188    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94189701_8e38_4499_855a_581b0e81a46c.slice/crio-313a8b5760e40f3a66bfa125c0b1bee27bd5a171a701eba8f85d4d8e2611774c WatchSource:0}: Error finding container 313a8b5760e40f3a66bfa125c0b1bee27bd5a171a701eba8f85d4d8e2611774c: Status 404 returned error can't find the container with id 313a8b5760e40f3a66bfa125c0b1bee27bd5a171a701eba8f85d4d8e2611774c
Sep 11 16:19:39.751347 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751307    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:39.751845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751596    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="prometheus" containerID="cri-o://c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22" gracePeriod=600
Sep 11 16:19:39.751845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751615    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1" gracePeriod=600
Sep 11 16:19:39.751845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751641    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="thanos-sidecar" containerID="cri-o://a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78" gracePeriod=600
Sep 11 16:19:39.751845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751684    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy" containerID="cri-o://bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8" gracePeriod=600
Sep 11 16:19:39.751845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751644    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="config-reloader" containerID="cri-o://a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8" gracePeriod=600
Sep 11 16:19:39.751845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.751641    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-web" containerID="cri-o://8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b" gracePeriod=600
Sep 11 16:19:39.799455 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.798771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4f95b6f-4vfrw" event={"ID":"94189701-8e38-4499-855a-581b0e81a46c","Type":"ContainerStarted","Data":"cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c"}
Sep 11 16:19:39.799455 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.798849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4f95b6f-4vfrw" event={"ID":"94189701-8e38-4499-855a-581b0e81a46c","Type":"ContainerStarted","Data":"313a8b5760e40f3a66bfa125c0b1bee27bd5a171a701eba8f85d4d8e2611774c"}
Sep 11 16:19:39.819785 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:39.819727    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79d4f95b6f-4vfrw" podStartSLOduration=1.8197086919999998 podStartE2EDuration="1.819708692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:39.818843686 +0000 UTC m=+155.499948112" watchObservedRunningTime="2026-09-11 16:19:39.819708692 +0000 UTC m=+155.500813116"
Sep 11 16:19:40.807214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807179    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1" exitCode=0
Sep 11 16:19:40.807214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807203    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8" exitCode=0
Sep 11 16:19:40.807214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807209    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78" exitCode=0
Sep 11 16:19:40.807214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807215    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8" exitCode=0
Sep 11 16:19:40.807214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807220    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22" exitCode=0
Sep 11 16:19:40.807780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1"}
Sep 11 16:19:40.807780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8"}
Sep 11 16:19:40.807780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78"}
Sep 11 16:19:40.807780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8"}
Sep 11 16:19:40.807780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:40.807501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22"}
Sep 11 16:19:41.013725 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.013696    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.137903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.137875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-thanos-prometheus-http-client-file\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.137903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.137907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-2\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.137940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-config-out\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.137973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-1\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.137998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-kubelet-serving-ca-bundle\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-0\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-serving-certs-ca-bundle\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-db\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-metrics-client-ca\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-kube-rbac-proxy\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-config\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm278\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-kube-api-access-bm278\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-trusted-ca-bundle\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-metrics-client-certs\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-web-config\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-grpc-tls\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138545    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-tls-assets\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-tls\") pod \"887cb074-4c71-4a3c-82d1-d5c628443f1d\" (UID: \"887cb074-4c71-4a3c-82d1-d5c628443f1d\") "
Sep 11 16:19:41.138849 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.139239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.138984    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.139239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.139003    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.139239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.139019    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.139239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.139021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.139239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.139034    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.140415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.140123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:41.140415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.140157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.140803 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.140777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-config" (OuterVolumeSpecName: "config") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.141415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.141175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.141415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.141173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:41.141577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.141456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-kube-api-access-bm278" (OuterVolumeSpecName: "kube-api-access-bm278") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "kube-api-access-bm278". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:41.142315 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.142281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.142788 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.142715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-config-out" (OuterVolumeSpecName: "config-out") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:41.143213 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.143185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.143697 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.143663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.143800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.143698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.144063 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.144032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.144151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.144129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.144218 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.144199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:41.155035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.155007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-web-config" (OuterVolumeSpecName: "web-config") pod "887cb074-4c71-4a3c-82d1-d5c628443f1d" (UID: "887cb074-4c71-4a3c-82d1-d5c628443f1d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240234    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-web-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240262    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-grpc-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240271    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-tls-assets\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240281    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240290    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240302    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-config-out\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240310    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240318    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240326    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/887cb074-4c71-4a3c-82d1-d5c628443f1d-prometheus-k8s-db\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240335    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240344    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240353    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/887cb074-4c71-4a3c-82d1-d5c628443f1d-configmap-metrics-client-ca\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240362    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-kube-rbac-proxy\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240371    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240380    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm278\" (UniqueName: \"kubernetes.io/projected/887cb074-4c71-4a3c-82d1-d5c628443f1d-kube-api-access-bm278\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.240653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.240388    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/887cb074-4c71-4a3c-82d1-d5c628443f1d-secret-metrics-client-certs\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:19:41.814612 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.814577    2580 generic.go:356] "Generic (PLEG): container finished" podID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerID="8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b" exitCode=0
Sep 11 16:19:41.814983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.814648    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.814983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.814661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b"}
Sep 11 16:19:41.814983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.814727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"887cb074-4c71-4a3c-82d1-d5c628443f1d","Type":"ContainerDied","Data":"d183e99e42c3f6318f3130e5fde71388764612676e0cc5b37cc44da8ee713c78"}
Sep 11 16:19:41.814983 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.814759    2580 scope.go:117] "RemoveContainer" containerID="920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1"
Sep 11 16:19:41.823415 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.823393    2580 scope.go:117] "RemoveContainer" containerID="bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8"
Sep 11 16:19:41.830331 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.830313    2580 scope.go:117] "RemoveContainer" containerID="8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b"
Sep 11 16:19:41.837257 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.837235    2580 scope.go:117] "RemoveContainer" containerID="a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78"
Sep 11 16:19:41.842790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.842767    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:41.845215 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.845202    2580 scope.go:117] "RemoveContainer" containerID="a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8"
Sep 11 16:19:41.849277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.849257    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:41.852602 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.852585    2580 scope.go:117] "RemoveContainer" containerID="c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22"
Sep 11 16:19:41.859597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.859579    2580 scope.go:117] "RemoveContainer" containerID="acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27"
Sep 11 16:19:41.866610 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.866429    2580 scope.go:117] "RemoveContainer" containerID="920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1"
Sep 11 16:19:41.866710 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.866688    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1\": container with ID starting with 920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1 not found: ID does not exist" containerID="920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1"
Sep 11 16:19:41.866806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.866717    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1"} err="failed to get container status \"920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1\": rpc error: code = NotFound desc = could not find container \"920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1\": container with ID starting with 920061ef1b776e9a078559479fb5e15443d01bc210459107db554b3e0709fbb1 not found: ID does not exist"
Sep 11 16:19:41.866806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.866737    2580 scope.go:117] "RemoveContainer" containerID="bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8"
Sep 11 16:19:41.867047 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.867029    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8\": container with ID starting with bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8 not found: ID does not exist" containerID="bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8"
Sep 11 16:19:41.867123 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867056    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8"} err="failed to get container status \"bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8\": rpc error: code = NotFound desc = could not find container \"bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8\": container with ID starting with bb8a3ddf518c076988346dad416d7ddb33fd3fad1c5c8ff096532a4429536fe8 not found: ID does not exist"
Sep 11 16:19:41.867123 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867081    2580 scope.go:117] "RemoveContainer" containerID="8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b"
Sep 11 16:19:41.867363 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.867347    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b\": container with ID starting with 8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b not found: ID does not exist" containerID="8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b"
Sep 11 16:19:41.867425 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867368    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b"} err="failed to get container status \"8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b\": rpc error: code = NotFound desc = could not find container \"8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b\": container with ID starting with 8f4eae2bdf9be2fe50e00ddf8121665cd662e79c18f31f1b4c2faf1ac3177c6b not found: ID does not exist"
Sep 11 16:19:41.867425 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867382    2580 scope.go:117] "RemoveContainer" containerID="a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78"
Sep 11 16:19:41.867613 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.867596    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78\": container with ID starting with a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78 not found: ID does not exist" containerID="a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78"
Sep 11 16:19:41.867653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867617    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78"} err="failed to get container status \"a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78\": rpc error: code = NotFound desc = could not find container \"a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78\": container with ID starting with a92b818cebf0d874609291907aa090c09bc9e6691871d3d6cfb9aec9fc53ad78 not found: ID does not exist"
Sep 11 16:19:41.867653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867630    2580 scope.go:117] "RemoveContainer" containerID="a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8"
Sep 11 16:19:41.867861 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.867845    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8\": container with ID starting with a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8 not found: ID does not exist" containerID="a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8"
Sep 11 16:19:41.867911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8"} err="failed to get container status \"a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8\": rpc error: code = NotFound desc = could not find container \"a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8\": container with ID starting with a215c703b0a03094f442932e133ee67d2e21c4df0db4f4122eb2370c4fadd7b8 not found: ID does not exist"
Sep 11 16:19:41.867911 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.867876    2580 scope.go:117] "RemoveContainer" containerID="c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22"
Sep 11 16:19:41.868095 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.868080    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22\": container with ID starting with c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22 not found: ID does not exist" containerID="c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22"
Sep 11 16:19:41.868134 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.868097    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22"} err="failed to get container status \"c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22\": rpc error: code = NotFound desc = could not find container \"c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22\": container with ID starting with c3921db06b211dd34c0b26399d2cd89f6929edc11a7a90684f29d71c50be2c22 not found: ID does not exist"
Sep 11 16:19:41.868134 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.868108    2580 scope.go:117] "RemoveContainer" containerID="acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27"
Sep 11 16:19:41.868322 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:19:41.868305    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27\": container with ID starting with acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27 not found: ID does not exist" containerID="acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27"
Sep 11 16:19:41.868364 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.868324    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27"} err="failed to get container status \"acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27\": rpc error: code = NotFound desc = could not find container \"acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27\": container with ID starting with acda81acc128e2886e013534e5b9399fb39a3a8fa8847bb809064ad87f5bcc27 not found: ID does not exist"
Sep 11 16:19:41.877934 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.877888    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:41.878189 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878177    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="init-config-reloader"
Sep 11 16:19:41.878233 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878191    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="init-config-reloader"
Sep 11 16:19:41.878233 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878212    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="thanos-sidecar"
Sep 11 16:19:41.878233 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878221    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="thanos-sidecar"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878234    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-web"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878240    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-web"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878250    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-thanos"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878255    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-thanos"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878262    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="prometheus"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878267    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="prometheus"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878274    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="config-reloader"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878279    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="config-reloader"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878289    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy"
Sep 11 16:19:41.878316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878293    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy"
Sep 11 16:19:41.878577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878345    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="prometheus"
Sep 11 16:19:41.878577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="thanos-sidecar"
Sep 11 16:19:41.878577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy"
Sep 11 16:19:41.878577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878366    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-thanos"
Sep 11 16:19:41.878577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="kube-rbac-proxy-web"
Sep 11 16:19:41.878577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.878376    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" containerName="config-reloader"
Sep 11 16:19:41.883832 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.883673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.886881 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.886861    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 16:19:41.887047 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887023    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 16:19:41.887327 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.886946    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 16:19:41.887327 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887097    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 16:19:41.887327 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.886988    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 16:19:41.887500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 16:19:41.887500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 16:19:41.887904 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2swnr\""
Sep 11 16:19:41.887998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 16:19:41.887998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887979    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 16:19:41.888102 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.887892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9tvrh8k42v65q\""
Sep 11 16:19:41.888161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.888096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 16:19:41.888205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.888182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 16:19:41.888255 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.888222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 16:19:41.891133 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.890932    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 16:19:41.892592 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.892574    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 16:19:41.899583 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.899559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:41.946659 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.946845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.946845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4755f6f6-794c-48a1-aee0-0f9b45e9718d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.946845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.946845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.946845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4755f6f6-794c-48a1-aee0-0f9b45e9718d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.946986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfns\" (UniqueName: \"kubernetes.io/projected/4755f6f6-794c-48a1-aee0-0f9b45e9718d-kube-api-access-xgfns\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-config\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.947403 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:41.947285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.047939 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.047899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.047939 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.047945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-config\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.047967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4755f6f6-794c-48a1-aee0-0f9b45e9718d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4755f6f6-794c-48a1-aee0-0f9b45e9718d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfns\" (UniqueName: \"kubernetes.io/projected/4755f6f6-794c-48a1-aee0-0f9b45e9718d-kube-api-access-xgfns\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.048859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.048600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.051051 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.051186 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.051186 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.051460 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-config\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.051988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.052079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.052140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.052615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.052791 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.052691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.053112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.052879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.053112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.053035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4755f6f6-794c-48a1-aee0-0f9b45e9718d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.053943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.053914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4755f6f6-794c-48a1-aee0-0f9b45e9718d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.054443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.054414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4755f6f6-794c-48a1-aee0-0f9b45e9718d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.054611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.054594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.054727 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.054710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4755f6f6-794c-48a1-aee0-0f9b45e9718d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.061752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.061733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfns\" (UniqueName: \"kubernetes.io/projected/4755f6f6-794c-48a1-aee0-0f9b45e9718d-kube-api-access-xgfns\") pod \"prometheus-k8s-0\" (UID: \"4755f6f6-794c-48a1-aee0-0f9b45e9718d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.194067 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.193960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:42.478853 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.478810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:42.479789 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:19:42.479763    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4755f6f6_794c_48a1_aee0_0f9b45e9718d.slice/crio-cd939bae4e7456e04b5bb5aef74204bca267187b8e08c43a4df757a53ad7fdba WatchSource:0}: Error finding container cd939bae4e7456e04b5bb5aef74204bca267187b8e08c43a4df757a53ad7fdba: Status 404 returned error can't find the container with id cd939bae4e7456e04b5bb5aef74204bca267187b8e08c43a4df757a53ad7fdba
Sep 11 16:19:42.821052 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.820960    2580 generic.go:356] "Generic (PLEG): container finished" podID="4755f6f6-794c-48a1-aee0-0f9b45e9718d" containerID="061f6967c22c6c79e4cf615ca910d1ae64a6a6b5dd9a3f4eeec45f37e000b9cf" exitCode=0
Sep 11 16:19:42.821441 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.821052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerDied","Data":"061f6967c22c6c79e4cf615ca910d1ae64a6a6b5dd9a3f4eeec45f37e000b9cf"}
Sep 11 16:19:42.821441 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.821123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"cd939bae4e7456e04b5bb5aef74204bca267187b8e08c43a4df757a53ad7fdba"}
Sep 11 16:19:42.942190 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:42.942163    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887cb074-4c71-4a3c-82d1-d5c628443f1d" path="/var/lib/kubelet/pods/887cb074-4c71-4a3c-82d1-d5c628443f1d/volumes"
Sep 11 16:19:43.103950 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.103896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:43.205193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.205143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:43.829314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.829280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"91d57d8ce7154be3fb9ff2f5e92015251d805f7f2a14e206464ee9d867c7cd82"}
Sep 11 16:19:43.829777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.829345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"7e06bca9d92512ce122cf921d8e749d5397fb10b7f05fd46c591b07f6569e0dd"}
Sep 11 16:19:43.829777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.829370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"64d7a4107c8980f608cad21fe1d55525c5e1ca1f8b1b9196cf5238f5bca3538d"}
Sep 11 16:19:43.829777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.829395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"bf02068bcf577461a8dda7c16d788c0384f2e60df5e5618d297ed9b2176e5c0d"}
Sep 11 16:19:43.829777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.829419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"d8942c4805bbe28082827924ddeaf3a7b703d5af1a5ef3d92846762f61d22634"}
Sep 11 16:19:43.829777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.829441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4755f6f6-794c-48a1-aee0-0f9b45e9718d","Type":"ContainerStarted","Data":"9691c6ac2289ca709fd8e42185e410799075406adfa1e766dd3db9c79e2b7e48"}
Sep 11 16:19:43.860582 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:43.860488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.841898782 podStartE2EDuration="2.860466377s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.567595ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:43.858113791 +0000 UTC m=+159.539218238" watchObservedRunningTime="2026-09-11 16:19:43.860466377 +0000 UTC m=+159.541570803"
Sep 11 16:19:47.194937 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:47.194892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:48.743529 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:48.743491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:48.743986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:48.743546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:48.749037 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:48.749013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:48.855245 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:48.855219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:19:48.945501 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:48.945469    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5789865dc8-mrxb2"]
Sep 11 16:19:49.065912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:49.065800    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:49.166346 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:49.166292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:49.267797 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:49.267752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.285019 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:59.284913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.402095 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:19:59.402052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:13.273581 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:13.273545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:13.964373 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:13.964334    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5789865dc8-mrxb2" podUID="05c59b15-448a-420d-84ed-f06405931ed7" containerName="console" containerID="cri-o://1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf" gracePeriod=15
Sep 11 16:20:14.213579 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.213557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:20:14.310379 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzvcc\" (UniqueName: \"kubernetes.io/projected/05c59b15-448a-420d-84ed-f06405931ed7-kube-api-access-dzvcc\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.310379 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310343    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-serving-cert\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.310379 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-console-config\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.310877 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-service-ca\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.310877 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-oauth-config\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.310877 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-oauth-serving-cert\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.310877 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-trusted-ca-bundle\") pod \"05c59b15-448a-420d-84ed-f06405931ed7\" (UID: \"05c59b15-448a-420d-84ed-f06405931ed7\") "
Sep 11 16:20:14.311078 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-console-config" (OuterVolumeSpecName: "console-config") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:14.311078 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.310970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-service-ca" (OuterVolumeSpecName: "service-ca") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:14.311168 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.311068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:14.311168 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.311127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:14.312501 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.312471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c59b15-448a-420d-84ed-f06405931ed7-kube-api-access-dzvcc" (OuterVolumeSpecName: "kube-api-access-dzvcc") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "kube-api-access-dzvcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:14.312602 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.312501    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:14.312602 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.312545    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05c59b15-448a-420d-84ed-f06405931ed7" (UID: "05c59b15-448a-420d-84ed-f06405931ed7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:14.412075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412045    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-trusted-ca-bundle\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.412075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412076    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvcc\" (UniqueName: \"kubernetes.io/projected/05c59b15-448a-420d-84ed-f06405931ed7-kube-api-access-dzvcc\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.412244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412089    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-serving-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.412244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412098    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-console-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.412244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412107    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-service-ca\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.412244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412115    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05c59b15-448a-420d-84ed-f06405931ed7-console-oauth-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.412244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.412123    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05c59b15-448a-420d-84ed-f06405931ed7-oauth-serving-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:20:14.957008 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.956977    2580 generic.go:356] "Generic (PLEG): container finished" podID="05c59b15-448a-420d-84ed-f06405931ed7" containerID="1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf" exitCode=0
Sep 11 16:20:14.957149 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.957022    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5789865dc8-mrxb2"
Sep 11 16:20:14.957149 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.957053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5789865dc8-mrxb2" event={"ID":"05c59b15-448a-420d-84ed-f06405931ed7","Type":"ContainerDied","Data":"1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf"}
Sep 11 16:20:14.957149 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.957100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5789865dc8-mrxb2" event={"ID":"05c59b15-448a-420d-84ed-f06405931ed7","Type":"ContainerDied","Data":"5c4b9f983617a7903a9b11220d9c840fb07aee5c3b1fc06fc5a730514dabef5e"}
Sep 11 16:20:14.957149 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.957119    2580 scope.go:117] "RemoveContainer" containerID="1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf"
Sep 11 16:20:14.965247 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.965018    2580 scope.go:117] "RemoveContainer" containerID="1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf"
Sep 11 16:20:14.965313 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:20:14.965260    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf\": container with ID starting with 1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf not found: ID does not exist" containerID="1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf"
Sep 11 16:20:14.965313 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.965283    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf"} err="failed to get container status \"1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf\": rpc error: code = NotFound desc = could not find container \"1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf\": container with ID starting with 1f589377fb6099e73cb2a7a885e73a749d9554c9bd877515a37bec0e7f4aefaf not found: ID does not exist"
Sep 11 16:20:14.975747 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.975725    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5789865dc8-mrxb2"]
Sep 11 16:20:14.978156 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:14.978136    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5789865dc8-mrxb2"]
Sep 11 16:20:16.943003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:16.941088    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c59b15-448a-420d-84ed-f06405931ed7" path="/var/lib/kubelet/pods/05c59b15-448a-420d-84ed-f06405931ed7/volumes"
Sep 11 16:20:38.822278 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:38.822238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:38.922942 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:38.922905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:42.194986 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:42.194928    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:42.210246 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:42.210212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:43.082903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:43.082873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:47.377752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:20:47.377714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:06.457941 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:06.457903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:06.558731 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:06.558695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:06.661797 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:06.661760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:33.924378 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:33.924289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:34.012517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:34.012475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:39.798202 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:39.798164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:39.877078 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:39.877045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:39.979167 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:39.979129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:40.145519 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:40.145484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:40.295523 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:40.295485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:40.400984 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:40.400903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:40.470940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:40.470910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:40.623726 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:40.623687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:40.822356 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:40.822272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:41.018113 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:41.018071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:41.159235 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:41.159206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:41.258726 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:41.258690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:44.302512 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:21:44.302466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:04.684425 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:22:04.684399    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:22:25.550487 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:22:25.550449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:32.615862 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:22:32.615801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:32.674612 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:22:32.674578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:48.840663 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:22:48.840631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:00.549871 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:00.549628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:00.657627 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:00.657587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:00.748525 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:00.748493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:00.942116 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:00.942080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:01.035426 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:01.035387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:01.337388 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:01.337295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:01.432260 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:01.432233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:09.531370 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:09.531332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:21.101787 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:21.101751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:21.153558 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:21.153492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:35.296840 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.296803    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6486675c9b-tdvjw"]
Sep 11 16:23:35.297229 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.297132    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c59b15-448a-420d-84ed-f06405931ed7" containerName="console"
Sep 11 16:23:35.297229 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.297143    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c59b15-448a-420d-84ed-f06405931ed7" containerName="console"
Sep 11 16:23:35.297229 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.297196    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c59b15-448a-420d-84ed-f06405931ed7" containerName="console"
Sep 11 16:23:35.300143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.300126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.306895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.306597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-trusted-ca-bundle\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.306895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.306642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-config\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.306895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.306722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-service-ca\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.307032 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.306927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-serving-cert\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.307032 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.306971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-oauth-serving-cert\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.307097 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.307048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-oauth-config\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.307097 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.307071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbvh\" (UniqueName: \"kubernetes.io/projected/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-kube-api-access-qnbvh\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.310723 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.310698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6486675c9b-tdvjw"]
Sep 11 16:23:35.364801 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.364766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:35.407914 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.407888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-oauth-config\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.407916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbvh\" (UniqueName: \"kubernetes.io/projected/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-kube-api-access-qnbvh\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.407947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-trusted-ca-bundle\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.407965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-config\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.407987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-service-ca\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.408057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-serving-cert\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.408084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-oauth-serving-cert\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408781 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.408757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-oauth-serving-cert\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408905 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.408757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-config\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.408905 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.408862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-service-ca\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.409023 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.408936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-trusted-ca-bundle\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.410522 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.410499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-oauth-config\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.410596 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.410499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-console-serving-cert\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.416104 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.416081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbvh\" (UniqueName: \"kubernetes.io/projected/e01ed241-b08e-4be5-8a25-c3b0b7cfa706-kube-api-access-qnbvh\") pod \"console-6486675c9b-tdvjw\" (UID: \"e01ed241-b08e-4be5-8a25-c3b0b7cfa706\") " pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.426729 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.426706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:35.496850 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.496819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:35.554387 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.554312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:35.609121 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.609101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:35.882192 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:35.882169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6486675c9b-tdvjw"]
Sep 11 16:23:35.884452 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:23:35.884428    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01ed241_b08e_4be5_8a25_c3b0b7cfa706.slice/crio-de911d43fa1e61bc0b40f16aaa94a27abe33da5811d325e770deab030241dd29 WatchSource:0}: Error finding container de911d43fa1e61bc0b40f16aaa94a27abe33da5811d325e770deab030241dd29: Status 404 returned error can't find the container with id de911d43fa1e61bc0b40f16aaa94a27abe33da5811d325e770deab030241dd29
Sep 11 16:23:36.106666 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:36.106626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:36.165001 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:36.164929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:36.759075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:36.759040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6486675c9b-tdvjw" event={"ID":"e01ed241-b08e-4be5-8a25-c3b0b7cfa706","Type":"ContainerStarted","Data":"69bc8e7402c41531b5fb24e973ebfccb05be1e4107aeeba89b19ae3e8dbab04b"}
Sep 11 16:23:36.759433 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:36.759102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6486675c9b-tdvjw" event={"ID":"e01ed241-b08e-4be5-8a25-c3b0b7cfa706","Type":"ContainerStarted","Data":"de911d43fa1e61bc0b40f16aaa94a27abe33da5811d325e770deab030241dd29"}
Sep 11 16:23:36.781785 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:36.781740    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6486675c9b-tdvjw" podStartSLOduration=1.78172488 podStartE2EDuration="1.78172488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:36.780020197 +0000 UTC m=+392.461124622" watchObservedRunningTime="2026-09-11 16:23:36.78172488 +0000 UTC m=+392.462829305"
Sep 11 16:23:45.610090 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:45.610052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:45.610517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:45.610377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:45.615052 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:45.615028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:45.798078 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:45.798051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6486675c9b-tdvjw"
Sep 11 16:23:45.845799 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:45.845766    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d4f95b6f-4vfrw"]
Sep 11 16:23:45.962618 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:45.962553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:46.061595 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:46.061560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:46.161275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:46.161246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:46.222756 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:23:46.222692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:10.866203 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:10.866125    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79d4f95b6f-4vfrw" podUID="94189701-8e38-4499-855a-581b0e81a46c" containerName="console" containerID="cri-o://cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c" gracePeriod=15
Sep 11 16:24:11.117083 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.117019    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:24:11.218306 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-oauth-serving-cert\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218482 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-serving-cert\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218482 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-trusted-ca-bundle\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-oauth-config\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnkzn\" (UniqueName: \"kubernetes.io/projected/94189701-8e38-4499-855a-581b0e81a46c-kube-api-access-tnkzn\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218708 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-console-config\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218708 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-service-ca\") pod \"94189701-8e38-4499-855a-581b0e81a46c\" (UID: \"94189701-8e38-4499-855a-581b0e81a46c\") "
Sep 11 16:24:11.218800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:24:11.218800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.218750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:24:11.219044 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.219007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-console-config" (OuterVolumeSpecName: "console-config") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:24:11.219106 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.219060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-service-ca" (OuterVolumeSpecName: "service-ca") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:24:11.219140 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.219122    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-trusted-ca-bundle\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.219174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.219141    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-console-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.219174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.219154    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-oauth-serving-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.220611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.220587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94189701-8e38-4499-855a-581b0e81a46c-kube-api-access-tnkzn" (OuterVolumeSpecName: "kube-api-access-tnkzn") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "kube-api-access-tnkzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:24:11.220889 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.220870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:24:11.220944 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.220890    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "94189701-8e38-4499-855a-581b0e81a46c" (UID: "94189701-8e38-4499-855a-581b0e81a46c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:24:11.319673 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.319645    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94189701-8e38-4499-855a-581b0e81a46c-service-ca\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.319673 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.319670    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-serving-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.319856 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.319682    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94189701-8e38-4499-855a-581b0e81a46c-console-oauth-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.319856 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.319695    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnkzn\" (UniqueName: \"kubernetes.io/projected/94189701-8e38-4499-855a-581b0e81a46c-kube-api-access-tnkzn\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:24:11.902061 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.902025    2580 generic.go:356] "Generic (PLEG): container finished" podID="94189701-8e38-4499-855a-581b0e81a46c" containerID="cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c" exitCode=0
Sep 11 16:24:11.902498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.902073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4f95b6f-4vfrw"
Sep 11 16:24:11.902498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.902113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4f95b6f-4vfrw" event={"ID":"94189701-8e38-4499-855a-581b0e81a46c","Type":"ContainerDied","Data":"cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c"}
Sep 11 16:24:11.902498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.902162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4f95b6f-4vfrw" event={"ID":"94189701-8e38-4499-855a-581b0e81a46c","Type":"ContainerDied","Data":"313a8b5760e40f3a66bfa125c0b1bee27bd5a171a701eba8f85d4d8e2611774c"}
Sep 11 16:24:11.902498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.902181    2580 scope.go:117] "RemoveContainer" containerID="cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c"
Sep 11 16:24:11.910699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.910684    2580 scope.go:117] "RemoveContainer" containerID="cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c"
Sep 11 16:24:11.911021 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:24:11.910994    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c\": container with ID starting with cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c not found: ID does not exist" containerID="cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c"
Sep 11 16:24:11.911083 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.911030    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c"} err="failed to get container status \"cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c\": rpc error: code = NotFound desc = could not find container \"cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c\": container with ID starting with cb847b5d8b89fb38c0c350f43aa4335e53c5fadb2740dd925da54784d99c042c not found: ID does not exist"
Sep 11 16:24:11.930271 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.930245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d4f95b6f-4vfrw"]
Sep 11 16:24:11.936615 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:11.936597    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79d4f95b6f-4vfrw"]
Sep 11 16:24:12.939603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:12.939569    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94189701-8e38-4499-855a-581b0e81a46c" path="/var/lib/kubelet/pods/94189701-8e38-4499-855a-581b0e81a46c/volumes"
Sep 11 16:24:21.543198 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:21.543150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:21.632232 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:21.632198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:21.745624 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:21.745588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:21.881560 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:21.881528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:21.981981 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:21.981953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:22.790593 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:22.790554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:22.892061 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:22.892025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:38.305297 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.305209    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-7nxrf"]
Sep 11 16:24:38.305720 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.305668    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94189701-8e38-4499-855a-581b0e81a46c" containerName="console"
Sep 11 16:24:38.305720 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.305687    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94189701-8e38-4499-855a-581b0e81a46c" containerName="console"
Sep 11 16:24:38.305852 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.305788    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94189701-8e38-4499-855a-581b0e81a46c" containerName="console"
Sep 11 16:24:38.308743 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.308721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.312099 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.312077    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 11 16:24:38.313294 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.313271    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:24:38.313404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.313271    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:24:38.313404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.313320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9m8qd\""
Sep 11 16:24:38.313404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.313322    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 11 16:24:38.318329 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.318311    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7nxrf"]
Sep 11 16:24:38.335701 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.335676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wk62\" (UniqueName: \"kubernetes.io/projected/219f8569-40dc-4d3d-b751-5c3dee1d2e20-kube-api-access-7wk62\") pod \"s3-init-7nxrf\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") " pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.335792 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.335720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f8569-40dc-4d3d-b751-5c3dee1d2e20-model-cache\") pod \"s3-init-7nxrf\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") " pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.436117 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.436088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wk62\" (UniqueName: \"kubernetes.io/projected/219f8569-40dc-4d3d-b751-5c3dee1d2e20-kube-api-access-7wk62\") pod \"s3-init-7nxrf\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") " pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.436239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.436142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f8569-40dc-4d3d-b751-5c3dee1d2e20-model-cache\") pod \"s3-init-7nxrf\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") " pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.436570 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.436551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f8569-40dc-4d3d-b751-5c3dee1d2e20-model-cache\") pod \"s3-init-7nxrf\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") " pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.446253 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.446227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wk62\" (UniqueName: \"kubernetes.io/projected/219f8569-40dc-4d3d-b751-5c3dee1d2e20-kube-api-access-7wk62\") pod \"s3-init-7nxrf\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") " pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.631926 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.631899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7nxrf"
Sep 11 16:24:38.901076 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.901051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7nxrf"]
Sep 11 16:24:38.902849 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:24:38.902801    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219f8569_40dc_4d3d_b751_5c3dee1d2e20.slice/crio-ba498ba24fca7e964e8b31c6302f3a457888a34b5c269f32ab9ff9722bafcbff WatchSource:0}: Error finding container ba498ba24fca7e964e8b31c6302f3a457888a34b5c269f32ab9ff9722bafcbff: Status 404 returned error can't find the container with id ba498ba24fca7e964e8b31c6302f3a457888a34b5c269f32ab9ff9722bafcbff
Sep 11 16:24:38.904854 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:38.904835    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:24:39.002018 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:39.001989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7nxrf" event={"ID":"219f8569-40dc-4d3d-b751-5c3dee1d2e20","Type":"ContainerStarted","Data":"ba498ba24fca7e964e8b31c6302f3a457888a34b5c269f32ab9ff9722bafcbff"}
Sep 11 16:24:44.027536 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:44.027499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7nxrf" event={"ID":"219f8569-40dc-4d3d-b751-5c3dee1d2e20","Type":"ContainerStarted","Data":"54bddecae880741a3f38edb3d38c25db7ab6c466f5246abb29ec4aae9b271963"}
Sep 11 16:24:46.038101 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:46.038067    2580 generic.go:356] "Generic (PLEG): container finished" podID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerID="54bddecae880741a3f38edb3d38c25db7ab6c466f5246abb29ec4aae9b271963" exitCode=0
Sep 11 16:24:46.038567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:46.038130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7nxrf" event={"ID":"219f8569-40dc-4d3d-b751-5c3dee1d2e20","Type":"ContainerDied","Data":"54bddecae880741a3f38edb3d38c25db7ab6c466f5246abb29ec4aae9b271963"}
Sep 11 16:24:51.063178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:51.063142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7nxrf" event={"ID":"219f8569-40dc-4d3d-b751-5c3dee1d2e20","Type":"ContainerStarted","Data":"a2c651db35ad600247087c39d9ee0aa156a0a679c025675475d3e3a9536b2568"}
Sep 11 16:24:51.081771 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:24:51.081712    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-7nxrf" podStartSLOduration=2.274508182 podStartE2EDuration="13.08169695s" totalImagesPullingTime="9.021568208s" totalInitContainerRuntime="1.78562056s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:51.080226126 +0000 UTC m=+466.761330551" watchObservedRunningTime="2026-09-11 16:24:51.08169695 +0000 UTC m=+466.762801376"
Sep 11 16:25:32.371041 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:32.370999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:56.327431 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:56.327398    2580 generic.go:356] "Generic (PLEG): container finished" podID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerID="a2c651db35ad600247087c39d9ee0aa156a0a679c025675475d3e3a9536b2568" exitCode=0
Sep 11 16:25:56.327431 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:56.327435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7nxrf" event={"ID":"219f8569-40dc-4d3d-b751-5c3dee1d2e20","Type":"ContainerDied","Data":"a2c651db35ad600247087c39d9ee0aa156a0a679c025675475d3e3a9536b2568"}
Sep 11 16:25:57.451100 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.451073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7nxrf"
Sep 11 16:25:57.526521 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.526484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wk62\" (UniqueName: \"kubernetes.io/projected/219f8569-40dc-4d3d-b751-5c3dee1d2e20-kube-api-access-7wk62\") pod \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") "
Sep 11 16:25:57.526683 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.526570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f8569-40dc-4d3d-b751-5c3dee1d2e20-model-cache\") pod \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\" (UID: \"219f8569-40dc-4d3d-b751-5c3dee1d2e20\") "
Sep 11 16:25:57.528676 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.528649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219f8569-40dc-4d3d-b751-5c3dee1d2e20-kube-api-access-7wk62" (OuterVolumeSpecName: "kube-api-access-7wk62") pod "219f8569-40dc-4d3d-b751-5c3dee1d2e20" (UID: "219f8569-40dc-4d3d-b751-5c3dee1d2e20"). InnerVolumeSpecName "kube-api-access-7wk62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:57.544646 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.544619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219f8569-40dc-4d3d-b751-5c3dee1d2e20-model-cache" (OuterVolumeSpecName: "model-cache") pod "219f8569-40dc-4d3d-b751-5c3dee1d2e20" (UID: "219f8569-40dc-4d3d-b751-5c3dee1d2e20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:25:57.628075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.628032    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219f8569-40dc-4d3d-b751-5c3dee1d2e20-model-cache\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:25:57.628075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:57.628075    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wk62\" (UniqueName: \"kubernetes.io/projected/219f8569-40dc-4d3d-b751-5c3dee1d2e20-kube-api-access-7wk62\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:25:58.337901 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:58.337872    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7nxrf"
Sep 11 16:25:58.338070 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:58.337896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7nxrf" event={"ID":"219f8569-40dc-4d3d-b751-5c3dee1d2e20","Type":"ContainerDied","Data":"ba498ba24fca7e964e8b31c6302f3a457888a34b5c269f32ab9ff9722bafcbff"}
Sep 11 16:25:58.338070 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:58.337933    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba498ba24fca7e964e8b31c6302f3a457888a34b5c269f32ab9ff9722bafcbff"
Sep 11 16:25:58.689544 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:58.689508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:59.259494 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:25:59.259466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:09.866301 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:09.866267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:10.071331 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.071292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:10.109669 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.109641    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"]
Sep 11 16:26:10.109994 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.109981    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerName="download-hf-model"
Sep 11 16:26:10.110060 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.109995    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerName="download-hf-model"
Sep 11 16:26:10.110060 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.110006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerName="s3-init"
Sep 11 16:26:10.110060 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.110011    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerName="s3-init"
Sep 11 16:26:10.110168 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.110069    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="219f8569-40dc-4d3d-b751-5c3dee1d2e20" containerName="s3-init"
Sep 11 16:26:10.113327 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.113308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.116129 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.116105    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:26:10.116251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.116173    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:26:10.116251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.116208    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 11 16:26:10.116251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.116223    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:26:10.116251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.116241    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 11 16:26:10.116461 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.116258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:26:10.117361 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.117341    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:26:10.124098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.124078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"]
Sep 11 16:26:10.129427 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.129401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.129538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.129450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls24m\" (UniqueName: \"kubernetes.io/projected/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-api-access-ls24m\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.129538 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.129496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.129634 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.129553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.129634 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.129587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.145203 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.145180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:10.205775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.205746    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"]
Sep 11 16:26:10.209188 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.209172    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.211624 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.211598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 11 16:26:10.214270 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.214246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:10.219043 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.219024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"]
Sep 11 16:26:10.230811 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.230783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.230912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.230815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.230912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.230852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfhc\" (UniqueName: \"kubernetes.io/projected/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-api-access-zrfhc\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.230912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.230870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.230912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.230889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.230912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.230910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.231075 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05b8b65-3ba2-4188-958a-7027969f3a93-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.231118 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.231166 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls24m\" (UniqueName: \"kubernetes.io/projected/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-api-access-ls24m\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.231218 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.231596 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.231685 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.231685 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.231667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.233367 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.233350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.240689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.240667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls24m\" (UniqueName: \"kubernetes.io/projected/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-api-access-ls24m\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.301953 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.301921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:10.332178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05b8b65-3ba2-4188-958a-7027969f3a93-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfhc\" (UniqueName: \"kubernetes.io/projected/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-api-access-zrfhc\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332488 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:26:10.332400    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:26:10.332488 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:26:10.332472    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls podName:a05b8b65-3ba2-4188-958a-7027969f3a93 nodeName:}" failed. No retries permitted until 2026-09-11 16:26:10.832452186 +0000 UTC m=+546.513556604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:26:10.332681 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05b8b65-3ba2-4188-958a-7027969f3a93-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332838 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.332908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.332891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.341163 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.341140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfhc\" (UniqueName: \"kubernetes.io/projected/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-api-access-zrfhc\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.425460 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.425376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:10.699287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.699263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"]
Sep 11 16:26:10.701784 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:26:10.701759    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8753fa9d_0482_4b1a_b2c2_4bd148b8b6d7.slice/crio-bdd4976b25ce24e54b2378c360da716dc726c2c47c02e35fcae987f6dc61f6cc WatchSource:0}: Error finding container bdd4976b25ce24e54b2378c360da716dc726c2c47c02e35fcae987f6dc61f6cc: Status 404 returned error can't find the container with id bdd4976b25ce24e54b2378c360da716dc726c2c47c02e35fcae987f6dc61f6cc
Sep 11 16:26:10.836069 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.836030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:10.836243 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:26:10.836217    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:26:10.836335 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:26:10.836323    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls podName:a05b8b65-3ba2-4188-958a-7027969f3a93 nodeName:}" failed. No retries permitted until 2026-09-11 16:26:11.836297685 +0000 UTC m=+547.517402104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:26:10.926357 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:10.926320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:11.390833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:11.390779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerStarted","Data":"4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c"}
Sep 11 16:26:11.391003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:11.390872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerStarted","Data":"bdd4976b25ce24e54b2378c360da716dc726c2c47c02e35fcae987f6dc61f6cc"}
Sep 11 16:26:11.845178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:11.845091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:11.847484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:11.847455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:12.019748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:12.019711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:12.291890 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:12.291866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"]
Sep 11 16:26:12.293532 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:26:12.293502    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05b8b65_3ba2_4188_958a_7027969f3a93.slice/crio-8e6fe047e50b16a66d925c82fd46ac6be65de2e24ddd192e1d60572a09ba1bcd WatchSource:0}: Error finding container 8e6fe047e50b16a66d925c82fd46ac6be65de2e24ddd192e1d60572a09ba1bcd: Status 404 returned error can't find the container with id 8e6fe047e50b16a66d925c82fd46ac6be65de2e24ddd192e1d60572a09ba1bcd
Sep 11 16:26:12.397274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:12.397224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerStarted","Data":"fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315"}
Sep 11 16:26:12.397492 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:12.397304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerStarted","Data":"8e6fe047e50b16a66d925c82fd46ac6be65de2e24ddd192e1d60572a09ba1bcd"}
Sep 11 16:26:15.411894 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:15.411856    2580 generic.go:356] "Generic (PLEG): container finished" podID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerID="4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c" exitCode=0
Sep 11 16:26:15.412440 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:15.411929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerDied","Data":"4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c"}
Sep 11 16:26:16.421318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:16.421230    2580 generic.go:356] "Generic (PLEG): container finished" podID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerID="fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315" exitCode=0
Sep 11 16:26:16.421318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:16.421287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerDied","Data":"fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315"}
Sep 11 16:26:17.312102 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:17.312065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:32.526187 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:32.526149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerStarted","Data":"30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216"}
Sep 11 16:26:32.528917 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:32.528890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerStarted","Data":"74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1"}
Sep 11 16:26:34.540848 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.540735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerStarted","Data":"b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386"}
Sep 11 16:26:34.541329 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.540856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:34.542888 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.542865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerStarted","Data":"b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789"}
Sep 11 16:26:34.543019 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.543005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:34.543081 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.543038    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:34.544505 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.544470    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:26:34.564161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.564121    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podStartSLOduration=3.131855819 podStartE2EDuration="24.564102875s" totalImagesPullingTime="17.629739053s" totalInitContainerRuntime="3.802508003s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:34.56341785 +0000 UTC m=+570.244522274" watchObservedRunningTime="2026-09-11 16:26:34.564102875 +0000 UTC m=+570.245207299"
Sep 11 16:26:34.585207 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:34.585148    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podStartSLOduration=1.98919999 podStartE2EDuration="24.58512791s" totalImagesPullingTime="18.63162751s" totalInitContainerRuntime="3.96430041s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:34.584764542 +0000 UTC m=+570.265868980" watchObservedRunningTime="2026-09-11 16:26:34.58512791 +0000 UTC m=+570.266232334"
Sep 11 16:26:35.546842 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:35.546800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:35.547248 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:35.546801    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:26:35.547976 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:35.547954    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:26:36.550576 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:36.550528    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:26:40.551205 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:40.551177    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:26:40.551675 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:40.551645    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:26:41.555395 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:41.555366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:26:41.555867 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:41.555840    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:26:50.552165 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:50.552121    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:26:51.556745 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:26:51.556706    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:27:00.552458 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:00.552419    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:27:01.556386 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:01.556348    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:27:10.551930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:10.551890    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:27:11.556089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:11.556046    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:27:20.551797 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:20.551752    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:27:21.555886 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:21.555849    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:27:30.551702 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:30.551615    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:27:31.556357 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:31.556316    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:27:40.552148 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:40.552105    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:27:41.556693 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:41.556650    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 11 16:27:50.553043 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.553011    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:27:50.695964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.695930    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"]
Sep 11 16:27:50.696203 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.696152    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container" containerID="cri-o://74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1" gracePeriod=30
Sep 11 16:27:50.696353 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.696234    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kube-rbac-proxy" containerID="cri-o://b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386" gracePeriod=30
Sep 11 16:27:50.697029 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.697013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:27:50.864405 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.864368    2580 generic.go:356] "Generic (PLEG): container finished" podID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerID="b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386" exitCode=2
Sep 11 16:27:50.864553 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:50.864435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerDied","Data":"b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386"}
Sep 11 16:27:51.551069 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:51.551022    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 11 16:27:56.550814 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:56.550763    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 11 16:27:58.243221 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.243195    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:27:58.325333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") pod \"a05b8b65-3ba2-4188-958a-7027969f3a93\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") "
Sep 11 16:27:58.325510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05b8b65-3ba2-4188-958a-7027969f3a93-kserve-provision-location\") pod \"a05b8b65-3ba2-4188-958a-7027969f3a93\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") "
Sep 11 16:27:58.325510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrfhc\" (UniqueName: \"kubernetes.io/projected/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-api-access-zrfhc\") pod \"a05b8b65-3ba2-4188-958a-7027969f3a93\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") "
Sep 11 16:27:58.325510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-rbac-proxy-sar-config\") pod \"a05b8b65-3ba2-4188-958a-7027969f3a93\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") "
Sep 11 16:27:58.325510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-cabundle-cert\") pod \"a05b8b65-3ba2-4188-958a-7027969f3a93\" (UID: \"a05b8b65-3ba2-4188-958a-7027969f3a93\") "
Sep 11 16:27:58.325843 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05b8b65-3ba2-4188-958a-7027969f3a93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a05b8b65-3ba2-4188-958a-7027969f3a93" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:58.325935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a05b8b65-3ba2-4188-958a-7027969f3a93" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:58.325935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.325915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a05b8b65-3ba2-4188-958a-7027969f3a93" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:58.327594 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.327573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a05b8b65-3ba2-4188-958a-7027969f3a93" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:58.327654 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.327606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-api-access-zrfhc" (OuterVolumeSpecName: "kube-api-access-zrfhc") pod "a05b8b65-3ba2-4188-958a-7027969f3a93" (UID: "a05b8b65-3ba2-4188-958a-7027969f3a93"). InnerVolumeSpecName "kube-api-access-zrfhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:58.427004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.426961    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:27:58.427004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.426998    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a05b8b65-3ba2-4188-958a-7027969f3a93-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:27:58.427004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.427009    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a05b8b65-3ba2-4188-958a-7027969f3a93-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:27:58.427245 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.427017    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a05b8b65-3ba2-4188-958a-7027969f3a93-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:27:58.427245 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.427027    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfhc\" (UniqueName: \"kubernetes.io/projected/a05b8b65-3ba2-4188-958a-7027969f3a93-kube-api-access-zrfhc\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:27:58.898303 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.898266    2580 generic.go:356] "Generic (PLEG): container finished" podID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerID="74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1" exitCode=0
Sep 11 16:27:58.898471 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.898320    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"
Sep 11 16:27:58.898471 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.898352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerDied","Data":"74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1"}
Sep 11 16:27:58.898471 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.898444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv" event={"ID":"a05b8b65-3ba2-4188-958a-7027969f3a93","Type":"ContainerDied","Data":"8e6fe047e50b16a66d925c82fd46ac6be65de2e24ddd192e1d60572a09ba1bcd"}
Sep 11 16:27:58.898592 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.898473    2580 scope.go:117] "RemoveContainer" containerID="b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386"
Sep 11 16:27:58.906974 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.906958    2580 scope.go:117] "RemoveContainer" containerID="74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1"
Sep 11 16:27:58.914155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.914137    2580 scope.go:117] "RemoveContainer" containerID="fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315"
Sep 11 16:27:58.921254 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.921237    2580 scope.go:117] "RemoveContainer" containerID="b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386"
Sep 11 16:27:58.921523 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:27:58.921504    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386\": container with ID starting with b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386 not found: ID does not exist" containerID="b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386"
Sep 11 16:27:58.921567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.921530    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386"} err="failed to get container status \"b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386\": rpc error: code = NotFound desc = could not find container \"b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386\": container with ID starting with b7bdc92adce2c0198df151fe0cc3ed27d43166a2838adce81f6722af4dea0386 not found: ID does not exist"
Sep 11 16:27:58.921567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.921546    2580 scope.go:117] "RemoveContainer" containerID="74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1"
Sep 11 16:27:58.921834 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:27:58.921797    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1\": container with ID starting with 74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1 not found: ID does not exist" containerID="74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1"
Sep 11 16:27:58.921920 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.921841    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1"} err="failed to get container status \"74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1\": rpc error: code = NotFound desc = could not find container \"74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1\": container with ID starting with 74f9bcf801b4b06d83ec64917975ced27f22b1f22702144e86088ba9650352c1 not found: ID does not exist"
Sep 11 16:27:58.921920 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.921865    2580 scope.go:117] "RemoveContainer" containerID="fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315"
Sep 11 16:27:58.922169 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:27:58.922142    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315\": container with ID starting with fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315 not found: ID does not exist" containerID="fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315"
Sep 11 16:27:58.922169 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.922174    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315"} err="failed to get container status \"fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315\": rpc error: code = NotFound desc = could not find container \"fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315\": container with ID starting with fc865245f4df25c383be0d6a8488064dcc1ee0b3a7eb1a065144dc0ae31b6315 not found: ID does not exist"
Sep 11 16:27:58.924551 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.924529    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"]
Sep 11 16:27:58.928394 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.928374    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-fx6tv"]
Sep 11 16:27:58.939768 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:58.939743    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" path="/var/lib/kubelet/pods/a05b8b65-3ba2-4188-958a-7027969f3a93/volumes"
Sep 11 16:27:59.883884 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:59.883852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"]
Sep 11 16:27:59.884341 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:59.884070    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" containerID="cri-o://30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216" gracePeriod=30
Sep 11 16:27:59.884341 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:59.884122    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kube-rbac-proxy" containerID="cri-o://b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789" gracePeriod=30
Sep 11 16:27:59.951170 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:27:59.951139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:00.028234 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028199    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"]
Sep 11 16:28:00.028627 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028608    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kube-rbac-proxy"
Sep 11 16:28:00.028717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028628    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kube-rbac-proxy"
Sep 11 16:28:00.028717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028647    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="storage-initializer"
Sep 11 16:28:00.028717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="storage-initializer"
Sep 11 16:28:00.028717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container"
Sep 11 16:28:00.028717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028674    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container"
Sep 11 16:28:00.029049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028782    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kube-rbac-proxy"
Sep 11 16:28:00.029049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.028797    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05b8b65-3ba2-4188-958a-7027969f3a93" containerName="kserve-container"
Sep 11 16:28:00.032031 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.032010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.034614 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.034596    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 11 16:28:00.042215 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.042190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"]
Sep 11 16:28:00.142189 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.142084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f22641-c59f-4387-a85b-10a01a0a6d7d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.142189 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.142132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.142417 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.142203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcb6\" (UniqueName: \"kubernetes.io/projected/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-api-access-9tcb6\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.142417 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.142277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.142417 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.142342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f22641-c59f-4387-a85b-10a01a0a6d7d-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.243341 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.243303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f22641-c59f-4387-a85b-10a01a0a6d7d-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.243528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.243366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f22641-c59f-4387-a85b-10a01a0a6d7d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.243528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.243475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.243528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.243524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcb6\" (UniqueName: \"kubernetes.io/projected/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-api-access-9tcb6\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.243700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.243562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.243700 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.243677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f22641-c59f-4387-a85b-10a01a0a6d7d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.244277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.244256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.244337 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.244256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.245794 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.245772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f22641-c59f-4387-a85b-10a01a0a6d7d-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.260438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.260413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcb6\" (UniqueName: \"kubernetes.io/projected/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-api-access-9tcb6\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.342102 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.342055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:00.547443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.547404    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 11 16:28:00.551758 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.551733    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:28:00.621398 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.621370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"]
Sep 11 16:28:00.622992 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:28:00.622965    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f22641_c59f_4387_a85b_10a01a0a6d7d.slice/crio-0f64234bb179259fb45ed67d62e25ed6d9fd2ff09a443c756b6b735ac13bbc91 WatchSource:0}: Error finding container 0f64234bb179259fb45ed67d62e25ed6d9fd2ff09a443c756b6b735ac13bbc91: Status 404 returned error can't find the container with id 0f64234bb179259fb45ed67d62e25ed6d9fd2ff09a443c756b6b735ac13bbc91
Sep 11 16:28:00.910713 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.910614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerStarted","Data":"aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4"}
Sep 11 16:28:00.910713 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.910699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerStarted","Data":"0f64234bb179259fb45ed67d62e25ed6d9fd2ff09a443c756b6b735ac13bbc91"}
Sep 11 16:28:00.912845 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.912810    2580 generic.go:356] "Generic (PLEG): container finished" podID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerID="b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789" exitCode=2
Sep 11 16:28:00.912962 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:00.912861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerDied","Data":"b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789"}
Sep 11 16:28:04.478511 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.478481    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:28:04.582480 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls24m\" (UniqueName: \"kubernetes.io/projected/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-api-access-ls24m\") pod \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") "
Sep 11 16:28:04.582480 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582458    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kserve-provision-location\") pod \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") "
Sep 11 16:28:04.582637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-proxy-tls\") pod \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") "
Sep 11 16:28:04.582637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-rbac-proxy-sar-config\") pod \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") "
Sep 11 16:28:04.582637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-cabundle-cert\") pod \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\" (UID: \"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7\") "
Sep 11 16:28:04.582899 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" (UID: "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:28:04.582977 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582896    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" (UID: "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:04.583015 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.582971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" (UID: "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:04.584543 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.584521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" (UID: "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:28:04.584631 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.584554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-api-access-ls24m" (OuterVolumeSpecName: "kube-api-access-ls24m") pod "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" (UID: "8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7"). InnerVolumeSpecName "kube-api-access-ls24m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:04.684252 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.684217    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:28:04.684252 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.684253    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:28:04.684437 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.684279    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:28:04.684437 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.684295    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls24m\" (UniqueName: \"kubernetes.io/projected/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kube-api-access-ls24m\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:28:04.684437 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.684309    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:28:04.932329 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.932292    2580 generic.go:356] "Generic (PLEG): container finished" podID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerID="30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216" exitCode=0
Sep 11 16:28:04.932500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.932340    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"
Sep 11 16:28:04.932500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.932370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerDied","Data":"30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216"}
Sep 11 16:28:04.932500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.932430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz" event={"ID":"8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7","Type":"ContainerDied","Data":"bdd4976b25ce24e54b2378c360da716dc726c2c47c02e35fcae987f6dc61f6cc"}
Sep 11 16:28:04.932500 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.932462    2580 scope.go:117] "RemoveContainer" containerID="b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789"
Sep 11 16:28:04.934308 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.934263    2580 generic.go:356] "Generic (PLEG): container finished" podID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerID="aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4" exitCode=0
Sep 11 16:28:04.942474 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.942172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerDied","Data":"aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4"}
Sep 11 16:28:04.943203 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.943174    2580 scope.go:117] "RemoveContainer" containerID="30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216"
Sep 11 16:28:04.949745 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.949718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:04.950961 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.950935    2580 scope.go:117] "RemoveContainer" containerID="4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c"
Sep 11 16:28:04.958863 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.958844    2580 scope.go:117] "RemoveContainer" containerID="b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789"
Sep 11 16:28:04.959226 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:28:04.959116    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789\": container with ID starting with b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789 not found: ID does not exist" containerID="b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789"
Sep 11 16:28:04.959226 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.959146    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789"} err="failed to get container status \"b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789\": rpc error: code = NotFound desc = could not find container \"b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789\": container with ID starting with b4979e70aa6984d73f035d686ba7b99fae019fb9bd3e35bff1648a9614868789 not found: ID does not exist"
Sep 11 16:28:04.959226 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.959162    2580 scope.go:117] "RemoveContainer" containerID="30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216"
Sep 11 16:28:04.959435 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:28:04.959419    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216\": container with ID starting with 30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216 not found: ID does not exist" containerID="30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216"
Sep 11 16:28:04.959498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.959437    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216"} err="failed to get container status \"30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216\": rpc error: code = NotFound desc = could not find container \"30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216\": container with ID starting with 30fe6f211d00451f83cdbca243cdd32f940546e9550090a5eddf7fe547a17216 not found: ID does not exist"
Sep 11 16:28:04.959498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.959454    2580 scope.go:117] "RemoveContainer" containerID="4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c"
Sep 11 16:28:04.959680 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:28:04.959659    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c\": container with ID starting with 4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c not found: ID does not exist" containerID="4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c"
Sep 11 16:28:04.959731 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.959685    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c"} err="failed to get container status \"4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c\": rpc error: code = NotFound desc = could not find container \"4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c\": container with ID starting with 4d536a489c412e57fa100bc4f72ac7e42fc15ef4c9a39ee9c7c9515d597ad78c not found: ID does not exist"
Sep 11 16:28:04.975204 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.975177    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"]
Sep 11 16:28:04.983531 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:04.983510    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-nn9cz"]
Sep 11 16:28:05.049127 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.049107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.080215 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080190    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"]
Sep 11 16:28:05.080545 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080529    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="storage-initializer"
Sep 11 16:28:05.080637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080548    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="storage-initializer"
Sep 11 16:28:05.080637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080570    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kube-rbac-proxy"
Sep 11 16:28:05.080637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080578    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kube-rbac-proxy"
Sep 11 16:28:05.080637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080616    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container"
Sep 11 16:28:05.080637 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080625    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container"
Sep 11 16:28:05.080895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080707    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kube-rbac-proxy"
Sep 11 16:28:05.080895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.080728    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" containerName="kserve-container"
Sep 11 16:28:05.083807 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.083786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.086802 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.086784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-77a5a-predictor-serving-cert\""
Sep 11 16:28:05.086936 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.086810    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-77a5a-kube-rbac-proxy-sar-config\""
Sep 11 16:28:05.095562 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.095542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"]
Sep 11 16:28:05.165812 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.165778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.188605 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.188531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85cx\" (UniqueName: \"kubernetes.io/projected/71865bc5-3811-4475-b927-c0e8e0766179-kube-api-access-j85cx\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.188605 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.188584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.188776 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.188650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71865bc5-3811-4475-b927-c0e8e0766179-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.188776 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.188758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-cabundle-cert\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.188875 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.188795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.202567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.202522    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"]
Sep 11 16:28:05.206086 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.206065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.208912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.208888    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-77a5a-predictor-serving-cert\""
Sep 11 16:28:05.209039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.208929    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-77a5a-kube-rbac-proxy-sar-config\""
Sep 11 16:28:05.219949 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.219917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"]
Sep 11 16:28:05.255126 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.255093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.290260 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0578848e-863b-4046-bf07-423d9c85edbb-proxy-tls\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.290400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-cabundle-cert\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.290400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8tw\" (UniqueName: \"kubernetes.io/projected/0578848e-863b-4046-bf07-423d9c85edbb-kube-api-access-qq8tw\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.290400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.290400 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0578848e-863b-4046-bf07-423d9c85edbb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.290633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85cx\" (UniqueName: \"kubernetes.io/projected/71865bc5-3811-4475-b927-c0e8e0766179-kube-api-access-j85cx\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.290633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.290633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.290633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.290633 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:28:05.290567    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-serving-cert: secret "isvc-sklearn-graph-raw-77a5a-predictor-serving-cert" not found
Sep 11 16:28:05.290633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71865bc5-3811-4475-b927-c0e8e0766179-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.290886 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:28:05.290681    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls podName:71865bc5-3811-4475-b927-c0e8e0766179 nodeName:}" failed. No retries permitted until 2026-09-11 16:28:05.790655592 +0000 UTC m=+661.471760011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls") pod "isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" (UID: "71865bc5-3811-4475-b927-c0e8e0766179") : secret "isvc-sklearn-graph-raw-77a5a-predictor-serving-cert" not found
Sep 11 16:28:05.291006 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-cabundle-cert\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.291050 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.290985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71865bc5-3811-4475-b927-c0e8e0766179-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.291112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.291094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.300718 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.300696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85cx\" (UniqueName: \"kubernetes.io/projected/71865bc5-3811-4475-b927-c0e8e0766179-kube-api-access-j85cx\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.382732 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.382698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.391918 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.391890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0578848e-863b-4046-bf07-423d9c85edbb-proxy-tls\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.391993 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.391960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8tw\" (UniqueName: \"kubernetes.io/projected/0578848e-863b-4046-bf07-423d9c85edbb-kube-api-access-qq8tw\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.391993 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.391987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0578848e-863b-4046-bf07-423d9c85edbb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.392105 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.392031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.392272 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.392235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.392440 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.392418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0578848e-863b-4046-bf07-423d9c85edbb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.392735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.392719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.392813 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.392796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.394298 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.394281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0578848e-863b-4046-bf07-423d9c85edbb-proxy-tls\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.399636 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.399609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8tw\" (UniqueName: \"kubernetes.io/projected/0578848e-863b-4046-bf07-423d9c85edbb-kube-api-access-qq8tw\") pod \"isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.522179 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.522094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:05.794775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.794748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"]
Sep 11 16:28:05.796457 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.796429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.797076 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:28:05.797045    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0578848e_863b_4046_bf07_423d9c85edbb.slice/crio-6fde9af3f97b1b9c6565349c4195e197364c59d98736a43c7a8af65dac90dd69 WatchSource:0}: Error finding container 6fde9af3f97b1b9c6565349c4195e197364c59d98736a43c7a8af65dac90dd69: Status 404 returned error can't find the container with id 6fde9af3f97b1b9c6565349c4195e197364c59d98736a43c7a8af65dac90dd69
Sep 11 16:28:05.798893 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.798871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls\") pod \"isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:05.942627 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.942594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerStarted","Data":"828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329"}
Sep 11 16:28:05.942800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.942666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerStarted","Data":"e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a"}
Sep 11 16:28:05.942952 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.942935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:05.944199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.944177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerStarted","Data":"6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3"}
Sep 11 16:28:05.944300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.944218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerStarted","Data":"6fde9af3f97b1b9c6565349c4195e197364c59d98736a43c7a8af65dac90dd69"}
Sep 11 16:28:05.963703 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.963657    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podStartSLOduration=2.240686562 podStartE2EDuration="5.963637905s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.722951343s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:05.96233881 +0000 UTC m=+661.643443248" watchObservedRunningTime="2026-09-11 16:28:05.963637905 +0000 UTC m=+661.644742331"
Sep 11 16:28:05.995909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:05.995880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:06.268099 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:06.268070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"]
Sep 11 16:28:06.269769 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:28:06.269745    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71865bc5_3811_4475_b927_c0e8e0766179.slice/crio-272289dee88970dc5b0c821794b42a53be7c1ad24c60865fc154b37515547049 WatchSource:0}: Error finding container 272289dee88970dc5b0c821794b42a53be7c1ad24c60865fc154b37515547049: Status 404 returned error can't find the container with id 272289dee88970dc5b0c821794b42a53be7c1ad24c60865fc154b37515547049
Sep 11 16:28:06.940705 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:06.940672    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7" path="/var/lib/kubelet/pods/8753fa9d-0482-4b1a-b2c2-4bd148b8b6d7/volumes"
Sep 11 16:28:06.949726 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:06.949694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerStarted","Data":"cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c"}
Sep 11 16:28:06.949870 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:06.949759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerStarted","Data":"272289dee88970dc5b0c821794b42a53be7c1ad24c60865fc154b37515547049"}
Sep 11 16:28:06.950219 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:06.950196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:06.951555 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:06.951534    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:07.157854 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:07.157809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:07.954214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:07.954178    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:09.966631 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:09.966601    2580 generic.go:356] "Generic (PLEG): container finished" podID="0578848e-863b-4046-bf07-423d9c85edbb" containerID="6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3" exitCode=0
Sep 11 16:28:09.966966 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:09.966669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerDied","Data":"6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3"}
Sep 11 16:28:09.968277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:09.968259    2580 generic.go:356] "Generic (PLEG): container finished" podID="71865bc5-3811-4475-b927-c0e8e0766179" containerID="cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c" exitCode=0
Sep 11 16:28:09.968353 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:09.968290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerDied","Data":"cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c"}
Sep 11 16:28:10.979266 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:10.979209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerStarted","Data":"e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4"}
Sep 11 16:28:10.979752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:10.979285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerStarted","Data":"93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428"}
Sep 11 16:28:10.979752 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:10.979703    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:10.979894 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:10.979763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:10.981525 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:10.981488    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:11.002466 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:11.002283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podStartSLOduration=2.402053621 podStartE2EDuration="6.002263009s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.600209388s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:11.001657612 +0000 UTC m=+666.682762038" watchObservedRunningTime="2026-09-11 16:28:11.002263009 +0000 UTC m=+666.683367437"
Sep 11 16:28:11.987241 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:11.986873    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:12.959302 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:12.959268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:28:12.959906 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:12.959876    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:16.994246 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:16.994174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:28:16.994735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:16.994695    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:18.294098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:18.294063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:22.960043 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:22.960005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:26.994607 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:26.994564    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:30.080526 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:30.080483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerStarted","Data":"4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a"}
Sep 11 16:28:30.080998 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:30.080574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerStarted","Data":"af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f"}
Sep 11 16:28:30.081231 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:30.081070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:30.081340 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:30.081320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:30.083239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:30.083197    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:28:30.101916 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:30.101803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podStartSLOduration=1.214427234 podStartE2EDuration="25.101788245s" totalImagesPullingTime="19.877070627s" totalInitContainerRuntime="4.010290384s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:30.100038766 +0000 UTC m=+685.781143191" watchObservedRunningTime="2026-09-11 16:28:30.101788245 +0000 UTC m=+685.782892669"
Sep 11 16:28:31.086997 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:31.086958    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:28:32.091229 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:32.091191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:28:32.959995 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:32.959950    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:36.994572 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:36.994526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:37.095722 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:37.095690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:28:37.096275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:37.096245    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:28:42.960630 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:42.960584    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:46.995616 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:46.995571    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:47.096242 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:47.096204    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:28:52.959844 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:52.959782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:28:53.226936 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:53.226852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:56.994956 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:56.994863    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:28:57.097089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:28:57.097050    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:29:02.960043 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:02.960002    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:29:06.994626 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:06.994579    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:29:07.097048 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:07.097005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:29:12.960597 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:12.960553    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:29:16.995133 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:16.995093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:29:17.096715 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:17.096675    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:29:19.935663 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:19.935622    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:29:26.994715 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:26.994674    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:29:27.096296 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:27.096248    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:29:29.936675 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:29.936647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:29:31.935564 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:31.935534    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:29:37.096806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:37.096775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:29:40.083908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:40.083876    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"]
Sep 11 16:29:40.084318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:40.084150    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container" containerID="cri-o://e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a" gracePeriod=30
Sep 11 16:29:40.084318 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:40.084193    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kube-rbac-proxy" containerID="cri-o://828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329" gracePeriod=30
Sep 11 16:29:40.111399 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:40.111369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:40.400395 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:40.400359    2580 generic.go:356] "Generic (PLEG): container finished" podID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerID="828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329" exitCode=2
Sep 11 16:29:40.400580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:40.400411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerDied","Data":"828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329"}
Sep 11 16:29:42.954273 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:42.954237    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 11 16:29:44.727515 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.727490    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:29:44.883893 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.883864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tcb6\" (UniqueName: \"kubernetes.io/projected/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-api-access-9tcb6\") pod \"12f22641-c59f-4387-a85b-10a01a0a6d7d\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") "
Sep 11 16:29:44.884074 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.883900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-cabundle-cert\") pod \"12f22641-c59f-4387-a85b-10a01a0a6d7d\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") "
Sep 11 16:29:44.884074 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.883942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f22641-c59f-4387-a85b-10a01a0a6d7d-kserve-provision-location\") pod \"12f22641-c59f-4387-a85b-10a01a0a6d7d\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") "
Sep 11 16:29:44.884074 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.883989    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-rbac-proxy-sar-config\") pod \"12f22641-c59f-4387-a85b-10a01a0a6d7d\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") "
Sep 11 16:29:44.884074 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.884024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f22641-c59f-4387-a85b-10a01a0a6d7d-proxy-tls\") pod \"12f22641-c59f-4387-a85b-10a01a0a6d7d\" (UID: \"12f22641-c59f-4387-a85b-10a01a0a6d7d\") "
Sep 11 16:29:44.884352 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.884327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f22641-c59f-4387-a85b-10a01a0a6d7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12f22641-c59f-4387-a85b-10a01a0a6d7d" (UID: "12f22641-c59f-4387-a85b-10a01a0a6d7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:29:44.884352 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.884341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12f22641-c59f-4387-a85b-10a01a0a6d7d" (UID: "12f22641-c59f-4387-a85b-10a01a0a6d7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:44.884451 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.884416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12f22641-c59f-4387-a85b-10a01a0a6d7d" (UID: "12f22641-c59f-4387-a85b-10a01a0a6d7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:44.886100 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.886078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f22641-c59f-4387-a85b-10a01a0a6d7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12f22641-c59f-4387-a85b-10a01a0a6d7d" (UID: "12f22641-c59f-4387-a85b-10a01a0a6d7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:44.886164 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.886095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-api-access-9tcb6" (OuterVolumeSpecName: "kube-api-access-9tcb6") pod "12f22641-c59f-4387-a85b-10a01a0a6d7d" (UID: "12f22641-c59f-4387-a85b-10a01a0a6d7d"). InnerVolumeSpecName "kube-api-access-9tcb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:29:44.984758 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.984730    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tcb6\" (UniqueName: \"kubernetes.io/projected/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-api-access-9tcb6\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:44.984758 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.984754    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:44.984940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.984764    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12f22641-c59f-4387-a85b-10a01a0a6d7d-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:44.984940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.984773    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12f22641-c59f-4387-a85b-10a01a0a6d7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:44.984940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:44.984782    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12f22641-c59f-4387-a85b-10a01a0a6d7d-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:45.110339 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.110307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:45.229475 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.229392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:45.300256 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.300210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:45.364735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.364699    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"]
Sep 11 16:29:45.365066 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365053    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container"
Sep 11 16:29:45.365112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365067    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container"
Sep 11 16:29:45.365112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365084    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kube-rbac-proxy"
Sep 11 16:29:45.365112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365089    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kube-rbac-proxy"
Sep 11 16:29:45.365112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365102    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="storage-initializer"
Sep 11 16:29:45.365112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365107    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="storage-initializer"
Sep 11 16:29:45.365265 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365157    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kserve-container"
Sep 11 16:29:45.365265 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.365165    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerName="kube-rbac-proxy"
Sep 11 16:29:45.366981 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.366951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:45.368537 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.368520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.370961 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.370934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 11 16:29:45.371065 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.371020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 11 16:29:45.378962 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.378942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"]
Sep 11 16:29:45.422437 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.422402    2580 generic.go:356] "Generic (PLEG): container finished" podID="12f22641-c59f-4387-a85b-10a01a0a6d7d" containerID="e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a" exitCode=0
Sep 11 16:29:45.422593 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.422484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"
Sep 11 16:29:45.422593 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.422487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerDied","Data":"e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a"}
Sep 11 16:29:45.422593 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.422559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9" event={"ID":"12f22641-c59f-4387-a85b-10a01a0a6d7d","Type":"ContainerDied","Data":"0f64234bb179259fb45ed67d62e25ed6d9fd2ff09a443c756b6b735ac13bbc91"}
Sep 11 16:29:45.422773 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.422599    2580 scope.go:117] "RemoveContainer" containerID="828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329"
Sep 11 16:29:45.431170 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.431144    2580 scope.go:117] "RemoveContainer" containerID="e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a"
Sep 11 16:29:45.439044 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.439010    2580 scope.go:117] "RemoveContainer" containerID="aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4"
Sep 11 16:29:45.440126 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.440107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"]
Sep 11 16:29:45.444420 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.444399    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-bfjd9"]
Sep 11 16:29:45.446914 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.446897    2580 scope.go:117] "RemoveContainer" containerID="828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329"
Sep 11 16:29:45.447182 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:29:45.447163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329\": container with ID starting with 828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329 not found: ID does not exist" containerID="828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329"
Sep 11 16:29:45.447231 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.447194    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329"} err="failed to get container status \"828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329\": rpc error: code = NotFound desc = could not find container \"828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329\": container with ID starting with 828f62f6ca85f700109b54eb9578ad17541a4e7c6231b70875d3d8ccd7802329 not found: ID does not exist"
Sep 11 16:29:45.447231 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.447212    2580 scope.go:117] "RemoveContainer" containerID="e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a"
Sep 11 16:29:45.447440 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:29:45.447419    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a\": container with ID starting with e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a not found: ID does not exist" containerID="e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a"
Sep 11 16:29:45.447512 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.447451    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a"} err="failed to get container status \"e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a\": rpc error: code = NotFound desc = could not find container \"e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a\": container with ID starting with e0539deb9765aebf16d51c0729b7b8a18791b7b87d1562bb9e89255378b4249a not found: ID does not exist"
Sep 11 16:29:45.447512 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.447475    2580 scope.go:117] "RemoveContainer" containerID="aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4"
Sep 11 16:29:45.447699 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:29:45.447685    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4\": container with ID starting with aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4 not found: ID does not exist" containerID="aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4"
Sep 11 16:29:45.447737 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.447702    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4"} err="failed to get container status \"aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4\": rpc error: code = NotFound desc = could not find container \"aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4\": container with ID starting with aa996f2dc577f9d631ae1d5629abeae2276c443006296bdf65367094288365d4 not found: ID does not exist"
Sep 11 16:29:45.489429 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.489374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.489524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.489445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.489524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.489476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4x9\" (UniqueName: \"kubernetes.io/projected/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-api-access-zb4x9\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.489524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.489511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.489641 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.489580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4x9\" (UniqueName: \"kubernetes.io/projected/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-api-access-zb4x9\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590738 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.590988 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.591064 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.590967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.592809 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.592791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.600518 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.600498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4x9\" (UniqueName: \"kubernetes.io/projected/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-api-access-zb4x9\") pod \"isvc-canary-stop-predictor-5cdcf6545-q7nb4\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.607801 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.607779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:45.679888 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.679856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:45.953527 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:45.953475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"]
Sep 11 16:29:45.955663 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:29:45.955635    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73dfaaa_50ac_4ce3_b47f_fc3b6be3db99.slice/crio-7b9c3553c64037dd5eb6d9aeabcc8f6ec3f7c40a2d06f6eb46c8b7c4a3113172 WatchSource:0}: Error finding container 7b9c3553c64037dd5eb6d9aeabcc8f6ec3f7c40a2d06f6eb46c8b7c4a3113172: Status 404 returned error can't find the container with id 7b9c3553c64037dd5eb6d9aeabcc8f6ec3f7c40a2d06f6eb46c8b7c4a3113172
Sep 11 16:29:46.429532 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:46.429499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerStarted","Data":"af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf"}
Sep 11 16:29:46.429709 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:46.429566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerStarted","Data":"7b9c3553c64037dd5eb6d9aeabcc8f6ec3f7c40a2d06f6eb46c8b7c4a3113172"}
Sep 11 16:29:46.939837 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:46.939798    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f22641-c59f-4387-a85b-10a01a0a6d7d" path="/var/lib/kubelet/pods/12f22641-c59f-4387-a85b-10a01a0a6d7d/volumes"
Sep 11 16:29:50.452349 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:50.452317    2580 generic.go:356] "Generic (PLEG): container finished" podID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerID="af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf" exitCode=0
Sep 11 16:29:50.452775 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:50.452361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerDied","Data":"af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf"}
Sep 11 16:29:51.461319 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:51.461279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerStarted","Data":"9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800"}
Sep 11 16:29:51.461806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:51.461357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerStarted","Data":"07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66"}
Sep 11 16:29:51.461806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:51.461601    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:51.480632 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:51.480581    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podStartSLOduration=2.603588734 podStartE2EDuration="6.480562574s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.87697384s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:51.47926978 +0000 UTC m=+767.160374205" watchObservedRunningTime="2026-09-11 16:29:51.480562574 +0000 UTC m=+767.161666983"
Sep 11 16:29:52.466036 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:52.466006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:52.466964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:52.466939    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:29:53.470116 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:53.470071    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:29:55.161135 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.161105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:55.295347 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.295318    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"]
Sep 11 16:29:55.295575 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.295548    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" containerID="cri-o://93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428" gracePeriod=30
Sep 11 16:29:55.295649 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.295594    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kube-rbac-proxy" containerID="cri-o://e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4" gracePeriod=30
Sep 11 16:29:55.332445 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.332413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:55.410522 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.410492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"]
Sep 11 16:29:55.410757 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.410732    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" containerID="cri-o://af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f" gracePeriod=30
Sep 11 16:29:55.410894 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.410841    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kube-rbac-proxy" containerID="cri-o://4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a" gracePeriod=30
Sep 11 16:29:55.481113 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.481076    2580 generic.go:356] "Generic (PLEG): container finished" podID="71865bc5-3811-4475-b927-c0e8e0766179" containerID="e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4" exitCode=2
Sep 11 16:29:55.481293 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.481145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerDied","Data":"e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4"}
Sep 11 16:29:55.505183 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.505154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:55.560284 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.560252    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"]
Sep 11 16:29:55.563717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.563696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.566603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.566579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-785db-kube-rbac-proxy-sar-config\""
Sep 11 16:29:55.566603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.566597    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-785db-predictor-serving-cert\""
Sep 11 16:29:55.575196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.575175    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"]
Sep 11 16:29:55.597912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.597871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:55.670172 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.670086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:55.672777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.672746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.672921 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.672839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.672921 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.672905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db46f2a-5bb4-4cf1-973c-a71b36c10725-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.673011 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.672936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.673011 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.672970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhz7\" (UniqueName: \"kubernetes.io/projected/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-api-access-hrhz7\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.674141 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.674120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"]
Sep 11 16:29:55.677707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.677685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.680340 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.680318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-785db-predictor-serving-cert\""
Sep 11 16:29:55.680438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.680369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-785db-kube-rbac-proxy-sar-config\""
Sep 11 16:29:55.687290 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.687264    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"]
Sep 11 16:29:55.758244 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.758206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:55.773380 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db46f2a-5bb4-4cf1-973c-a71b36c10725-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.773493 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.773493 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b3a6e2-a597-42c3-b967-7d97ddd8700a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.773493 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhz7\" (UniqueName: \"kubernetes.io/projected/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-api-access-hrhz7\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.773707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.773707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.773707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mzsx\" (UniqueName: \"kubernetes.io/projected/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-api-access-8mzsx\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.773707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.773707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.773975 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.773754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.774029 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.774005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.774204 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.774178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.774411 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.774387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.775903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.775883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db46f2a-5bb4-4cf1-973c-a71b36c10725-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.782095 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.782070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhz7\" (UniqueName: \"kubernetes.io/projected/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-api-access-hrhz7\") pod \"isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.874301 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.874264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:29:55.874575 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.874551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mzsx\" (UniqueName: \"kubernetes.io/projected/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-api-access-8mzsx\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.874664 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.874601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.874724 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.874683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.874780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.874726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b3a6e2-a597-42c3-b967-7d97ddd8700a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.874859 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.874781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.875079 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.875054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.875479 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.875451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.875693 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.875674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.877247 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.877226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b3a6e2-a597-42c3-b967-7d97ddd8700a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.883518 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.883495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mzsx\" (UniqueName: \"kubernetes.io/projected/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-api-access-8mzsx\") pod \"isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:55.988373 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:55.988339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:29:56.152070 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.152046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"]
Sep 11 16:29:56.153996 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:29:56.153969    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db46f2a_5bb4_4cf1_973c_a71b36c10725.slice/crio-19d9cd077cfa495bbd9c5a83f2403563574db4017941092c1a322e6011042975 WatchSource:0}: Error finding container 19d9cd077cfa495bbd9c5a83f2403563574db4017941092c1a322e6011042975: Status 404 returned error can't find the container with id 19d9cd077cfa495bbd9c5a83f2403563574db4017941092c1a322e6011042975
Sep 11 16:29:56.302079 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.302055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"]
Sep 11 16:29:56.304133 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:29:56.304108    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b3a6e2_a597_42c3_b967_7d97ddd8700a.slice/crio-214d965f1cf31e18f30d40e2eb0ae628bb2c192a542f1389412150432d800816 WatchSource:0}: Error finding container 214d965f1cf31e18f30d40e2eb0ae628bb2c192a542f1389412150432d800816: Status 404 returned error can't find the container with id 214d965f1cf31e18f30d40e2eb0ae628bb2c192a542f1389412150432d800816
Sep 11 16:29:56.491653 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.491550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerStarted","Data":"ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60"}
Sep 11 16:29:56.491817 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.491675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerStarted","Data":"19d9cd077cfa495bbd9c5a83f2403563574db4017941092c1a322e6011042975"}
Sep 11 16:29:56.494080 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.494051    2580 generic.go:356] "Generic (PLEG): container finished" podID="0578848e-863b-4046-bf07-423d9c85edbb" containerID="4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a" exitCode=2
Sep 11 16:29:56.494209 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.494124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerDied","Data":"4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a"}
Sep 11 16:29:56.495890 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.495864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerStarted","Data":"a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841"}
Sep 11 16:29:56.496007 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.495915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerStarted","Data":"214d965f1cf31e18f30d40e2eb0ae628bb2c192a542f1389412150432d800816"}
Sep 11 16:29:56.987406 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:56.987369    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 11 16:29:57.092331 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:57.092285    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 11 16:29:57.096629 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:57.096603    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 11 16:29:58.476003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:58.475975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:29:58.476589 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:58.476560    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:29:59.156460 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.156432    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:29:59.206740 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.206706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq8tw\" (UniqueName: \"kubernetes.io/projected/0578848e-863b-4046-bf07-423d9c85edbb-kube-api-access-qq8tw\") pod \"0578848e-863b-4046-bf07-423d9c85edbb\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") "
Sep 11 16:29:59.206943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.206773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-kube-rbac-proxy-sar-config\") pod \"0578848e-863b-4046-bf07-423d9c85edbb\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") "
Sep 11 16:29:59.206943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.206797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-cabundle-cert\") pod \"0578848e-863b-4046-bf07-423d9c85edbb\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") "
Sep 11 16:29:59.206943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.206871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0578848e-863b-4046-bf07-423d9c85edbb-kserve-provision-location\") pod \"0578848e-863b-4046-bf07-423d9c85edbb\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") "
Sep 11 16:29:59.206943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.206897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0578848e-863b-4046-bf07-423d9c85edbb-proxy-tls\") pod \"0578848e-863b-4046-bf07-423d9c85edbb\" (UID: \"0578848e-863b-4046-bf07-423d9c85edbb\") "
Sep 11 16:29:59.207162 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.207126    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0578848e-863b-4046-bf07-423d9c85edbb" (UID: "0578848e-863b-4046-bf07-423d9c85edbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:59.207211 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.207170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0578848e-863b-4046-bf07-423d9c85edbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0578848e-863b-4046-bf07-423d9c85edbb" (UID: "0578848e-863b-4046-bf07-423d9c85edbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:29:59.207252 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.207222    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0578848e-863b-4046-bf07-423d9c85edbb" (UID: "0578848e-863b-4046-bf07-423d9c85edbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:59.207292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.207249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:59.207292 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.207262    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0578848e-863b-4046-bf07-423d9c85edbb-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:59.208766 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.208746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0578848e-863b-4046-bf07-423d9c85edbb-kube-api-access-qq8tw" (OuterVolumeSpecName: "kube-api-access-qq8tw") pod "0578848e-863b-4046-bf07-423d9c85edbb" (UID: "0578848e-863b-4046-bf07-423d9c85edbb"). InnerVolumeSpecName "kube-api-access-qq8tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:29:59.208892 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.208875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0578848e-863b-4046-bf07-423d9c85edbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0578848e-863b-4046-bf07-423d9c85edbb" (UID: "0578848e-863b-4046-bf07-423d9c85edbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:59.307865 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.307769    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8tw\" (UniqueName: \"kubernetes.io/projected/0578848e-863b-4046-bf07-423d9c85edbb-kube-api-access-qq8tw\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:59.307865 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.307797    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0578848e-863b-4046-bf07-423d9c85edbb-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:59.307865 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.307807    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0578848e-863b-4046-bf07-423d9c85edbb-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:29:59.512198 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.512165    2580 generic.go:356] "Generic (PLEG): container finished" podID="0578848e-863b-4046-bf07-423d9c85edbb" containerID="af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f" exitCode=0
Sep 11 16:29:59.512623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.512220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"
Sep 11 16:29:59.512623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.512250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerDied","Data":"af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f"}
Sep 11 16:29:59.512623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.512308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h" event={"ID":"0578848e-863b-4046-bf07-423d9c85edbb","Type":"ContainerDied","Data":"6fde9af3f97b1b9c6565349c4195e197364c59d98736a43c7a8af65dac90dd69"}
Sep 11 16:29:59.512623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.512336    2580 scope.go:117] "RemoveContainer" containerID="4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a"
Sep 11 16:29:59.521578 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.521562    2580 scope.go:117] "RemoveContainer" containerID="af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f"
Sep 11 16:29:59.531051 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.530964    2580 scope.go:117] "RemoveContainer" containerID="6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3"
Sep 11 16:29:59.537913 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.537882    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"]
Sep 11 16:29:59.541162 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.541139    2580 scope.go:117] "RemoveContainer" containerID="4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a"
Sep 11 16:29:59.543236 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:29:59.543203    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a\": container with ID starting with 4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a not found: ID does not exist" containerID="4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a"
Sep 11 16:29:59.543348 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.543246    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a"} err="failed to get container status \"4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a\": rpc error: code = NotFound desc = could not find container \"4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a\": container with ID starting with 4865214b094c39ba7ef1f37ab19826319939cecd1a6c6f4d1157245bde404e4a not found: ID does not exist"
Sep 11 16:29:59.543348 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.543272    2580 scope.go:117] "RemoveContainer" containerID="af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f"
Sep 11 16:29:59.543631 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:29:59.543593    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f\": container with ID starting with af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f not found: ID does not exist" containerID="af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f"
Sep 11 16:29:59.543756 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.543642    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f"} err="failed to get container status \"af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f\": rpc error: code = NotFound desc = could not find container \"af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f\": container with ID starting with af8cf8459ae7b65047c799963c9a9ef7c7a78443200a72124359f6c764034a5f not found: ID does not exist"
Sep 11 16:29:59.543756 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.543665    2580 scope.go:117] "RemoveContainer" containerID="6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3"
Sep 11 16:29:59.543954 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.543859    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-77a5a-predictor-6f4b6b69df-g475h"]
Sep 11 16:29:59.544094 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:29:59.544069    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3\": container with ID starting with 6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3 not found: ID does not exist" containerID="6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3"
Sep 11 16:29:59.544149 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:29:59.544099    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3"} err="failed to get container status \"6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3\": rpc error: code = NotFound desc = could not find container \"6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3\": container with ID starting with 6d625c21e7f146002a5ffb3803921685c2de3a239641b8bb89e04adf633a2da3 not found: ID does not exist"
Sep 11 16:30:00.519094 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:00.519060    2580 generic.go:356] "Generic (PLEG): container finished" podID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerID="a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841" exitCode=0
Sep 11 16:30:00.519510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:00.519134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerDied","Data":"a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841"}
Sep 11 16:30:00.520816 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:00.520796    2580 generic.go:356] "Generic (PLEG): container finished" podID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerID="ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60" exitCode=0
Sep 11 16:30:00.520914 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:00.520848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerDied","Data":"ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60"}
Sep 11 16:30:00.940276 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:00.940244    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0578848e-863b-4046-bf07-423d9c85edbb" path="/var/lib/kubelet/pods/0578848e-863b-4046-bf07-423d9c85edbb/volumes"
Sep 11 16:30:01.527978 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.527939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerStarted","Data":"d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e"}
Sep 11 16:30:01.528453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.528012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerStarted","Data":"700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8"}
Sep 11 16:30:01.528453 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.528238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:30:01.530011 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.529988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerStarted","Data":"ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68"}
Sep 11 16:30:01.530124 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.530032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerStarted","Data":"c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b"}
Sep 11 16:30:01.530348 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.530331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:30:01.530419 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.530372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:30:01.531635 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.531608    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:01.550320 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.550268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podStartSLOduration=2.713943119 podStartE2EDuration="6.550252499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83630938s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:30:01.548379595 +0000 UTC m=+777.229484023" watchObservedRunningTime="2026-09-11 16:30:01.550252499 +0000 UTC m=+777.231356923"
Sep 11 16:30:01.566921 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.566877    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podStartSLOduration=2.966637791 podStartE2EDuration="6.566862682s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.600224891s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:30:01.565303976 +0000 UTC m=+777.246408402" watchObservedRunningTime="2026-09-11 16:30:01.566862682 +0000 UTC m=+777.247967106"
Sep 11 16:30:01.935533 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.935491    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 11 16:30:01.987693 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:01.987647    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 11 16:30:02.535256 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:02.535217    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:02.535713 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:02.535263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:30:02.536440 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:02.536413    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:03.539323 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:03.539283    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:04.041155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.041132    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:30:04.153611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-kube-rbac-proxy-sar-config\") pod \"71865bc5-3811-4475-b927-c0e8e0766179\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") "
Sep 11 16:30:04.153611 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-cabundle-cert\") pod \"71865bc5-3811-4475-b927-c0e8e0766179\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") "
Sep 11 16:30:04.153842 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71865bc5-3811-4475-b927-c0e8e0766179-kserve-provision-location\") pod \"71865bc5-3811-4475-b927-c0e8e0766179\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") "
Sep 11 16:30:04.153842 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls\") pod \"71865bc5-3811-4475-b927-c0e8e0766179\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") "
Sep 11 16:30:04.153842 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j85cx\" (UniqueName: \"kubernetes.io/projected/71865bc5-3811-4475-b927-c0e8e0766179-kube-api-access-j85cx\") pod \"71865bc5-3811-4475-b927-c0e8e0766179\" (UID: \"71865bc5-3811-4475-b927-c0e8e0766179\") "
Sep 11 16:30:04.154011 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153939    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71865bc5-3811-4475-b927-c0e8e0766179" (UID: "71865bc5-3811-4475-b927-c0e8e0766179"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:04.154011 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.153981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71865bc5-3811-4475-b927-c0e8e0766179-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71865bc5-3811-4475-b927-c0e8e0766179" (UID: "71865bc5-3811-4475-b927-c0e8e0766179"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:30:04.154120 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.154017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71865bc5-3811-4475-b927-c0e8e0766179" (UID: "71865bc5-3811-4475-b927-c0e8e0766179"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:04.154120 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.154081    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:30:04.154120 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.154100    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71865bc5-3811-4475-b927-c0e8e0766179-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:30:04.155796 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.155767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71865bc5-3811-4475-b927-c0e8e0766179" (UID: "71865bc5-3811-4475-b927-c0e8e0766179"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:30:04.155796 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.155774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71865bc5-3811-4475-b927-c0e8e0766179-kube-api-access-j85cx" (OuterVolumeSpecName: "kube-api-access-j85cx") pod "71865bc5-3811-4475-b927-c0e8e0766179" (UID: "71865bc5-3811-4475-b927-c0e8e0766179"). InnerVolumeSpecName "kube-api-access-j85cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:30:04.254675 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.254645    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71865bc5-3811-4475-b927-c0e8e0766179-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:30:04.254675 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.254670    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71865bc5-3811-4475-b927-c0e8e0766179-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:30:04.254675 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.254680    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j85cx\" (UniqueName: \"kubernetes.io/projected/71865bc5-3811-4475-b927-c0e8e0766179-kube-api-access-j85cx\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:30:04.545964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.545872    2580 generic.go:356] "Generic (PLEG): container finished" podID="71865bc5-3811-4475-b927-c0e8e0766179" containerID="93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428" exitCode=0
Sep 11 16:30:04.545964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.545935    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"
Sep 11 16:30:04.545964 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.545947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerDied","Data":"93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428"}
Sep 11 16:30:04.546510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.546002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl" event={"ID":"71865bc5-3811-4475-b927-c0e8e0766179","Type":"ContainerDied","Data":"272289dee88970dc5b0c821794b42a53be7c1ad24c60865fc154b37515547049"}
Sep 11 16:30:04.546510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.546029    2580 scope.go:117] "RemoveContainer" containerID="e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4"
Sep 11 16:30:04.554710 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.554693    2580 scope.go:117] "RemoveContainer" containerID="93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428"
Sep 11 16:30:04.563710 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.563688    2580 scope.go:117] "RemoveContainer" containerID="cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c"
Sep 11 16:30:04.575002 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.574978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"]
Sep 11 16:30:04.575110 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.575008    2580 scope.go:117] "RemoveContainer" containerID="e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4"
Sep 11 16:30:04.575351 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:30:04.575332    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4\": container with ID starting with e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4 not found: ID does not exist" containerID="e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4"
Sep 11 16:30:04.575410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.575359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4"} err="failed to get container status \"e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4\": rpc error: code = NotFound desc = could not find container \"e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4\": container with ID starting with e506d020245b51a443bc93e6f826426fd426dce54b25ca91a6cf8cef827d02f4 not found: ID does not exist"
Sep 11 16:30:04.575410 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.575376    2580 scope.go:117] "RemoveContainer" containerID="93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428"
Sep 11 16:30:04.575639 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:30:04.575619    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428\": container with ID starting with 93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428 not found: ID does not exist" containerID="93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428"
Sep 11 16:30:04.575711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.575649    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428"} err="failed to get container status \"93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428\": rpc error: code = NotFound desc = could not find container \"93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428\": container with ID starting with 93de0b0f78c558fbc03220d610b7c66a9baf878af993f60526c947ca76cbe428 not found: ID does not exist"
Sep 11 16:30:04.575711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.575673    2580 scope.go:117] "RemoveContainer" containerID="cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c"
Sep 11 16:30:04.575946 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:30:04.575928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c\": container with ID starting with cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c not found: ID does not exist" containerID="cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c"
Sep 11 16:30:04.575996 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.575950    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c"} err="failed to get container status \"cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c\": rpc error: code = NotFound desc = could not find container \"cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c\": container with ID starting with cfaf7104fc50d1cdebf3e20a9d823d3e96753eac1e9457ff07313a21be72d43c not found: ID does not exist"
Sep 11 16:30:04.576735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.576717    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-77a5a-predictor-889c756d4-fcqxl"]
Sep 11 16:30:04.940454 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:04.940423    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71865bc5-3811-4475-b927-c0e8e0766179" path="/var/lib/kubelet/pods/71865bc5-3811-4475-b927-c0e8e0766179/volumes"
Sep 11 16:30:07.540003 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:07.539974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:30:07.540641 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:07.540613    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:08.476848 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:08.476798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:30:08.544162 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:08.544130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:30:08.544669 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:08.544641    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:17.541332 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:17.541285    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:18.476492 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:18.476450    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:30:18.545598 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:18.545553    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:27.541224 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:27.541139    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:28.476731 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:28.476679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:30:28.545299 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:28.545264    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:37.541266 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:37.541227    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:38.477248 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:38.477207    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:30:38.545046 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:38.545004    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:47.541532 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:47.541492    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:48.476661 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:48.476621    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:30:48.545467 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:48.545433    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:30:57.540782 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:57.540734    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:30:58.477539 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:58.477483    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:30:58.544887 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:30:58.544850    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:31:06.940443 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:06.940416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:31:07.541860 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:07.541807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:31:08.544688 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:08.544653    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:31:10.935265 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:10.935225    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:31:15.273451 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.273410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:15.292573 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.292537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"]
Sep 11 16:31:15.292793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.292763    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" containerID="cri-o://07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66" gracePeriod=30
Sep 11 16:31:15.292903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.292800    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kube-rbac-proxy" containerID="cri-o://9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800" gracePeriod=30
Sep 11 16:31:15.335799 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.335766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:15.849105 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.849075    2580 generic.go:356] "Generic (PLEG): container finished" podID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerID="9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800" exitCode=2
Sep 11 16:31:15.849274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:15.849119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerDied","Data":"9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800"}
Sep 11 16:31:16.936297 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:16.936252    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:31:18.471352 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:18.471311    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 11 16:31:20.128341 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.128311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:31:20.143156 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-proxy-tls\") pod \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") "
Sep 11 16:31:20.143297 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kserve-provision-location\") pod \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") "
Sep 11 16:31:20.143297 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-rbac-proxy-sar-config\") pod \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") "
Sep 11 16:31:20.143297 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-cabundle-cert\") pod \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") "
Sep 11 16:31:20.143456 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb4x9\" (UniqueName: \"kubernetes.io/projected/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-api-access-zb4x9\") pod \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\" (UID: \"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99\") "
Sep 11 16:31:20.143590 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" (UID: "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:31:20.143687 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" (UID: "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:20.143731 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.143670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" (UID: "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:20.145289 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.145267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" (UID: "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:31:20.145403 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.145383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-api-access-zb4x9" (OuterVolumeSpecName: "kube-api-access-zb4x9") pod "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" (UID: "f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99"). InnerVolumeSpecName "kube-api-access-zb4x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:20.244974 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.244898    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.244974 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.244925    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.244974 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.244936    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.244974 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.244947    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.244974 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.244956    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4x9\" (UniqueName: \"kubernetes.io/projected/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99-kube-api-access-zb4x9\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.871981 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.871945    2580 generic.go:356] "Generic (PLEG): container finished" podID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerID="07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66" exitCode=0
Sep 11 16:31:20.872178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.871996    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"
Sep 11 16:31:20.872178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.871998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerDied","Data":"07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66"}
Sep 11 16:31:20.872178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.872128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4" event={"ID":"f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99","Type":"ContainerDied","Data":"7b9c3553c64037dd5eb6d9aeabcc8f6ec3f7c40a2d06f6eb46c8b7c4a3113172"}
Sep 11 16:31:20.872178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.872169    2580 scope.go:117] "RemoveContainer" containerID="9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800"
Sep 11 16:31:20.880689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.880671    2580 scope.go:117] "RemoveContainer" containerID="07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66"
Sep 11 16:31:20.887927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.887908    2580 scope.go:117] "RemoveContainer" containerID="af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf"
Sep 11 16:31:20.894105 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.894084    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"]
Sep 11 16:31:20.895416 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.895401    2580 scope.go:117] "RemoveContainer" containerID="9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800"
Sep 11 16:31:20.895670 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:20.895651    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800\": container with ID starting with 9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800 not found: ID does not exist" containerID="9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800"
Sep 11 16:31:20.895718 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.895680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800"} err="failed to get container status \"9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800\": rpc error: code = NotFound desc = could not find container \"9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800\": container with ID starting with 9179e15ae9de9c2cf55f7f0520daf251c809e3f0267bf1dd5d1a9f0781bdb800 not found: ID does not exist"
Sep 11 16:31:20.895718 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.895703    2580 scope.go:117] "RemoveContainer" containerID="07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66"
Sep 11 16:31:20.896177 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:20.896153    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66\": container with ID starting with 07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66 not found: ID does not exist" containerID="07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66"
Sep 11 16:31:20.896287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.896184    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66"} err="failed to get container status \"07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66\": rpc error: code = NotFound desc = could not find container \"07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66\": container with ID starting with 07fac6808b5cecae21c423327c2052e2a96bf7d84f6073d1d5970bd995972e66 not found: ID does not exist"
Sep 11 16:31:20.896287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.896229    2580 scope.go:117] "RemoveContainer" containerID="af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf"
Sep 11 16:31:20.896609 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:20.896586    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf\": container with ID starting with af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf not found: ID does not exist" containerID="af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf"
Sep 11 16:31:20.896679 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.896614    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf"} err="failed to get container status \"af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf\": rpc error: code = NotFound desc = could not find container \"af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf\": container with ID starting with af59d84ef0f2471306c991882ba7bcfd5bd34eda5006272588783e93cfa341cf not found: ID does not exist"
Sep 11 16:31:20.898320 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.898301    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-q7nb4"]
Sep 11 16:31:20.939519 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.939491    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" path="/var/lib/kubelet/pods/f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99/volumes"
Sep 11 16:31:20.939973 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:20.939954    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:31:25.500262 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:25.500226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:25.567680 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:25.567652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:35.411624 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:35.411586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:35.493650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:35.493615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:45.487918 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.487884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:45.564548 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.564517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"]
Sep 11 16:31:45.564776 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.564755    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container" containerID="cri-o://700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8" gracePeriod=30
Sep 11 16:31:45.564879 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.564854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kube-rbac-proxy" containerID="cri-o://d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e" gracePeriod=30
Sep 11 16:31:45.594435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.594411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:45.662742 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.662712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:45.712062 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.712030    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"]
Sep 11 16:31:45.712272 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.712249    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" containerID="cri-o://c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b" gracePeriod=30
Sep 11 16:31:45.712345 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.712292    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kube-rbac-proxy" containerID="cri-o://ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68" gracePeriod=30
Sep 11 16:31:45.736296 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.736272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:45.975025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.974993    2580 generic.go:356] "Generic (PLEG): container finished" podID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerID="ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68" exitCode=2
Sep 11 16:31:45.975185 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.975062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerDied","Data":"ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68"}
Sep 11 16:31:45.976979 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.976955    2580 generic.go:356] "Generic (PLEG): container finished" podID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerID="d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e" exitCode=2
Sep 11 16:31:45.977105 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:45.976995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerDied","Data":"d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e"}
Sep 11 16:31:46.021509 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:46.021479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:47.535633 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:47.535587    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 11 16:31:47.540982 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:47.540951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:31:48.540153 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:48.540110    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 11 16:31:49.462720 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.462694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:31:49.610760 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.610673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kserve-provision-location\") pod \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") "
Sep 11 16:31:49.610760 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.610715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mzsx\" (UniqueName: \"kubernetes.io/projected/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-api-access-8mzsx\") pod \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") "
Sep 11 16:31:49.610760 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.610754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-rbac-proxy-sar-config\") pod \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") "
Sep 11 16:31:49.611314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.610814    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-cabundle-cert\") pod \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") "
Sep 11 16:31:49.611314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.610851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b3a6e2-a597-42c3-b967-7d97ddd8700a-proxy-tls\") pod \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\" (UID: \"45b3a6e2-a597-42c3-b967-7d97ddd8700a\") "
Sep 11 16:31:49.611314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.611140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45b3a6e2-a597-42c3-b967-7d97ddd8700a" (UID: "45b3a6e2-a597-42c3-b967-7d97ddd8700a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:49.611314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.611153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45b3a6e2-a597-42c3-b967-7d97ddd8700a" (UID: "45b3a6e2-a597-42c3-b967-7d97ddd8700a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:31:49.611314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.611239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45b3a6e2-a597-42c3-b967-7d97ddd8700a" (UID: "45b3a6e2-a597-42c3-b967-7d97ddd8700a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:49.612800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.612774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b3a6e2-a597-42c3-b967-7d97ddd8700a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45b3a6e2-a597-42c3-b967-7d97ddd8700a" (UID: "45b3a6e2-a597-42c3-b967-7d97ddd8700a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:31:49.612800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.612795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-api-access-8mzsx" (OuterVolumeSpecName: "kube-api-access-8mzsx") pod "45b3a6e2-a597-42c3-b967-7d97ddd8700a" (UID: "45b3a6e2-a597-42c3-b967-7d97ddd8700a"). InnerVolumeSpecName "kube-api-access-8mzsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:49.712198 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.712166    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:49.712198 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.712196    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b3a6e2-a597-42c3-b967-7d97ddd8700a-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:49.712198 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.712205    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:49.712432 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.712214    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mzsx\" (UniqueName: \"kubernetes.io/projected/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-api-access-8mzsx\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:49.712432 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.712224    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b3a6e2-a597-42c3-b967-7d97ddd8700a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:49.997303 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.997270    2580 generic.go:356] "Generic (PLEG): container finished" podID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerID="c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b" exitCode=0
Sep 11 16:31:49.997474 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.997342    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"
Sep 11 16:31:49.997474 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.997356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerDied","Data":"c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b"}
Sep 11 16:31:49.997474 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.997416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn" event={"ID":"45b3a6e2-a597-42c3-b967-7d97ddd8700a","Type":"ContainerDied","Data":"214d965f1cf31e18f30d40e2eb0ae628bb2c192a542f1389412150432d800816"}
Sep 11 16:31:49.997474 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:49.997445    2580 scope.go:117] "RemoveContainer" containerID="ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68"
Sep 11 16:31:50.005868 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.005852    2580 scope.go:117] "RemoveContainer" containerID="c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b"
Sep 11 16:31:50.013033 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.013011    2580 scope.go:117] "RemoveContainer" containerID="a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841"
Sep 11 16:31:50.019866 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.019845    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"]
Sep 11 16:31:50.020543 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.020525    2580 scope.go:117] "RemoveContainer" containerID="ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68"
Sep 11 16:31:50.020814 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:50.020796    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68\": container with ID starting with ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68 not found: ID does not exist" containerID="ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68"
Sep 11 16:31:50.020901 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.020845    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68"} err="failed to get container status \"ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68\": rpc error: code = NotFound desc = could not find container \"ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68\": container with ID starting with ef3507ba2d86e34d68f08ccde29ff28bcc4f45cf875eb94dcce37cb09651ac68 not found: ID does not exist"
Sep 11 16:31:50.020901 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.020868    2580 scope.go:117] "RemoveContainer" containerID="c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b"
Sep 11 16:31:50.021132 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:50.021115    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b\": container with ID starting with c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b not found: ID does not exist" containerID="c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b"
Sep 11 16:31:50.021208 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.021137    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b"} err="failed to get container status \"c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b\": rpc error: code = NotFound desc = could not find container \"c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b\": container with ID starting with c5419e689c9ccb5db339f978deeab3705aeb82d8fd3220947794eba32f04ab5b not found: ID does not exist"
Sep 11 16:31:50.021208 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.021156    2580 scope.go:117] "RemoveContainer" containerID="a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841"
Sep 11 16:31:50.021458 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:50.021410    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841\": container with ID starting with a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841 not found: ID does not exist" containerID="a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841"
Sep 11 16:31:50.021615 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.021463    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841"} err="failed to get container status \"a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841\": rpc error: code = NotFound desc = could not find container \"a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841\": container with ID starting with a52882fc69c78354c23a252a908c390e1fd0bc4e953f1be70ad8e84167b60841 not found: ID does not exist"
Sep 11 16:31:50.023659 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.023638    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-785db-predictor-867fc895d9-qfrkn"]
Sep 11 16:31:50.309282 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.309256    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:31:50.417595 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.417559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db46f2a-5bb4-4cf1-973c-a71b36c10725-proxy-tls\") pod \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") "
Sep 11 16:31:50.417790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.417602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrhz7\" (UniqueName: \"kubernetes.io/projected/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-api-access-hrhz7\") pod \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") "
Sep 11 16:31:50.417790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.417648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kserve-provision-location\") pod \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") "
Sep 11 16:31:50.417790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.417757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-rbac-proxy-sar-config\") pod \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") "
Sep 11 16:31:50.418006 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.417795    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-cabundle-cert\") pod \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\" (UID: \"4db46f2a-5bb4-4cf1-973c-a71b36c10725\") "
Sep 11 16:31:50.418184 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.418095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4db46f2a-5bb4-4cf1-973c-a71b36c10725" (UID: "4db46f2a-5bb4-4cf1-973c-a71b36c10725"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:31:50.418337 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.418158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4db46f2a-5bb4-4cf1-973c-a71b36c10725" (UID: "4db46f2a-5bb4-4cf1-973c-a71b36c10725"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:50.418337 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.418212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4db46f2a-5bb4-4cf1-973c-a71b36c10725" (UID: "4db46f2a-5bb4-4cf1-973c-a71b36c10725"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:50.419790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.419763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db46f2a-5bb4-4cf1-973c-a71b36c10725-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4db46f2a-5bb4-4cf1-973c-a71b36c10725" (UID: "4db46f2a-5bb4-4cf1-973c-a71b36c10725"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:31:50.419940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.419811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-api-access-hrhz7" (OuterVolumeSpecName: "kube-api-access-hrhz7") pod "4db46f2a-5bb4-4cf1-973c-a71b36c10725" (UID: "4db46f2a-5bb4-4cf1-973c-a71b36c10725"). InnerVolumeSpecName "kube-api-access-hrhz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:50.519435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.519399    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db46f2a-5bb4-4cf1-973c-a71b36c10725-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:50.519435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.519429    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhz7\" (UniqueName: \"kubernetes.io/projected/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-api-access-hrhz7\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:50.519435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.519439    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:50.519679 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.519448    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:50.519679 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.519457    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db46f2a-5bb4-4cf1-973c-a71b36c10725-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:31:50.940226 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:50.940192    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" path="/var/lib/kubelet/pods/45b3a6e2-a597-42c3-b967-7d97ddd8700a/volumes"
Sep 11 16:31:51.005069 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.005039    2580 generic.go:356] "Generic (PLEG): container finished" podID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerID="700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8" exitCode=0
Sep 11 16:31:51.005236 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.005104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerDied","Data":"700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8"}
Sep 11 16:31:51.005236 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.005117    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"
Sep 11 16:31:51.005236 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.005154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb" event={"ID":"4db46f2a-5bb4-4cf1-973c-a71b36c10725","Type":"ContainerDied","Data":"19d9cd077cfa495bbd9c5a83f2403563574db4017941092c1a322e6011042975"}
Sep 11 16:31:51.005236 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.005182    2580 scope.go:117] "RemoveContainer" containerID="d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e"
Sep 11 16:31:51.013844 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.013810    2580 scope.go:117] "RemoveContainer" containerID="700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8"
Sep 11 16:31:51.021412 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.021395    2580 scope.go:117] "RemoveContainer" containerID="ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60"
Sep 11 16:31:51.025173 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.025148    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"]
Sep 11 16:31:51.029627 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.029606    2580 scope.go:117] "RemoveContainer" containerID="d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e"
Sep 11 16:31:51.029965 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:51.029940    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e\": container with ID starting with d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e not found: ID does not exist" containerID="d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e"
Sep 11 16:31:51.030047 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.029975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e"} err="failed to get container status \"d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e\": rpc error: code = NotFound desc = could not find container \"d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e\": container with ID starting with d8ff5c3afcfe2ad6c29fff4f2976b90906ec7862f4b26d19f2d59fd64ec6b85e not found: ID does not exist"
Sep 11 16:31:51.030047 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.030000    2580 scope.go:117] "RemoveContainer" containerID="700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8"
Sep 11 16:31:51.030292 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:51.030274    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8\": container with ID starting with 700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8 not found: ID does not exist" containerID="700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8"
Sep 11 16:31:51.030345 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.030298    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8"} err="failed to get container status \"700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8\": rpc error: code = NotFound desc = could not find container \"700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8\": container with ID starting with 700426927a35bd9f8080624c76afce32d1f85569f57c1ece1b01413fa9a13fa8 not found: ID does not exist"
Sep 11 16:31:51.030345 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.030315    2580 scope.go:117] "RemoveContainer" containerID="ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60"
Sep 11 16:31:51.030584 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:31:51.030568    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60\": container with ID starting with ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60 not found: ID does not exist" containerID="ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60"
Sep 11 16:31:51.030648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.030590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60"} err="failed to get container status \"ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60\": rpc error: code = NotFound desc = could not find container \"ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60\": container with ID starting with ad920fba2694ac9eb4b7e77dc5fc8ebdaa8146ef6ea2962ac5b723d33b3c3a60 not found: ID does not exist"
Sep 11 16:31:51.031055 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:51.031038    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-785db-predictor-5bb558968d-792kb"]
Sep 11 16:31:52.939873 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:31:52.939840    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" path="/var/lib/kubelet/pods/4db46f2a-5bb4-4cf1-973c-a71b36c10725/volumes"
Sep 11 16:32:05.675020 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.674988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:05.711686 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.711644    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"]
Sep 11 16:32:05.712055 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712038    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712057    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712072    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="storage-initializer"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712080    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="storage-initializer"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712091    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712099    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712110    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712119    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712130    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712139    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712151 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712148    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712157    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712168    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712176    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712186    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712207    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712234    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712252    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712272    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712280    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712295    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712304    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712321    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712329    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="storage-initializer"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712337    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712345    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712423    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712436    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712448    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712457    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0578848e-863b-4046-bf07-423d9c85edbb" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712465    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712473    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kserve-container"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712480    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71865bc5-3811-4475-b927-c0e8e0766179" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712490    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b3a6e2-a597-42c3-b967-7d97ddd8700a" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712500    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db46f2a-5bb4-4cf1-973c-a71b36c10725" containerName="kube-rbac-proxy"
Sep 11 16:32:05.712650 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.712509    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73dfaaa-50ac-4ce3-b47f-fc3b6be3db99" containerName="kube-rbac-proxy"
Sep 11 16:32:05.717593 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.717572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.720339 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-ecc5e-predictor-serving-cert\""
Sep 11 16:32:05.720339 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:32:05.720514 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720391    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:32:05.720514 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720393    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:32:05.720514 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:32:05.720514 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720431    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-ecc5e-kube-rbac-proxy-sar-config\""
Sep 11 16:32:05.720784 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.720649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:32:05.726547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.726522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"]
Sep 11 16:32:05.748056 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.748022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmls\" (UniqueName: \"kubernetes.io/projected/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-api-access-7pmls\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.748165 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.748075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-cabundle-cert\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.748165 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.748145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.748249 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.748180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e180306-c3db-4197-9a4c-7810ee0d7434-proxy-tls\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.748249 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.748210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e180306-c3db-4197-9a4c-7810ee0d7434-kserve-provision-location\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.808563 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.808530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:05.848887 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.848854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmls\" (UniqueName: \"kubernetes.io/projected/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-api-access-7pmls\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849045 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.848915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-cabundle-cert\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849045 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.848963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849045 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.848998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e180306-c3db-4197-9a4c-7810ee0d7434-proxy-tls\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849045 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.849034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e180306-c3db-4197-9a4c-7810ee0d7434-kserve-provision-location\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849445 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.849421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e180306-c3db-4197-9a4c-7810ee0d7434-kserve-provision-location\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849649 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.849629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.849703 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.849663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-cabundle-cert\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.851330 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.851313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e180306-c3db-4197-9a4c-7810ee0d7434-proxy-tls\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:05.857300 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:05.857277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmls\" (UniqueName: \"kubernetes.io/projected/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-api-access-7pmls\") pod \"isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:06.029394 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:06.029308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:06.301861 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:06.301837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"]
Sep 11 16:32:06.303772 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:32:06.303742    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e180306_c3db_4197_9a4c_7810ee0d7434.slice/crio-c6a896e9a6183695cc73f4f33bf755a836a1eb9f6a9f36848e7be88564c2c69c WatchSource:0}: Error finding container c6a896e9a6183695cc73f4f33bf755a836a1eb9f6a9f36848e7be88564c2c69c: Status 404 returned error can't find the container with id c6a896e9a6183695cc73f4f33bf755a836a1eb9f6a9f36848e7be88564c2c69c
Sep 11 16:32:07.076107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:07.076068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerStarted","Data":"387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e"}
Sep 11 16:32:07.076506 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:07.076135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerStarted","Data":"c6a896e9a6183695cc73f4f33bf755a836a1eb9f6a9f36848e7be88564c2c69c"}
Sep 11 16:32:10.090658 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:10.090623    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerID="387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e" exitCode=0
Sep 11 16:32:10.091027 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:10.090697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerDied","Data":"387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e"}
Sep 11 16:32:10.205038 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:10.205021    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:32:11.106711 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:11.106665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerStarted","Data":"3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391"}
Sep 11 16:32:11.107190 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:11.106760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerStarted","Data":"584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60"}
Sep 11 16:32:13.117726 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:13.117691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerStarted","Data":"42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac"}
Sep 11 16:32:13.118141 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:13.117936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:13.118141 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:13.118068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:13.119407 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:13.119378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:13.142897 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:13.142853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podStartSLOduration=2.383523883 podStartE2EDuration="8.142836119s" totalImagesPullingTime="2.221587949s" totalInitContainerRuntime="3.537724287s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:13.140166211 +0000 UTC m=+908.821270637" watchObservedRunningTime="2026-09-11 16:32:13.142836119 +0000 UTC m=+908.823940537"
Sep 11 16:32:14.122138 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:14.122103    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:14.122607 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:14.122191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:14.123255 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:14.123230    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:15.126613 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:15.126571    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:15.127092 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:15.127003    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:15.130171 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:15.130149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:32:16.130778 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:16.130743    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:16.131203 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:16.131069    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:26.131581 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:26.131528    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:26.132096 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:26.132074    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:36.130777 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:36.130731    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:36.132214 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:36.131210    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:46.131187 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:46.131141    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:46.131656 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:46.131539    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:56.130888 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:56.130811    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:32:56.131344 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:32:56.131301    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:06.130737 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:06.130691    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:33:06.131253 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:06.131059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:10.678249 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.678215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:10.704723 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.704683    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"]
Sep 11 16:33:10.707279 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.707261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.710047 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.710027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d91b1e-predictor-serving-cert\""
Sep 11 16:33:10.710152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.710047    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d91b1e\""
Sep 11 16:33:10.710152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.710141    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d91b1e-dockercfg-jrq74\""
Sep 11 16:33:10.710325 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.710305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d91b1e-kube-rbac-proxy-sar-config\""
Sep 11 16:33:10.719772 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.719749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"]
Sep 11 16:33:10.812735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.812690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.812909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.812740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c670fabd-99b5-48fc-a356-a88c9c460d29-kserve-provision-location\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.812909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.812787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.812909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.812814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvdp\" (UniqueName: \"kubernetes.io/projected/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-api-access-tlvdp\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.812909 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.812891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-cabundle-cert\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.913329 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.913296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.913496 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.913337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvdp\" (UniqueName: \"kubernetes.io/projected/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-api-access-tlvdp\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.913496 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.913361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-cabundle-cert\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.913496 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.913423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.913496 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.913440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c670fabd-99b5-48fc-a356-a88c9c460d29-kserve-provision-location\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.913710 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:33:10.913557    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-serving-cert: secret "isvc-secondary-d91b1e-predictor-serving-cert" not found
Sep 11 16:33:10.913710 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:33:10.913622    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls podName:c670fabd-99b5-48fc-a356-a88c9c460d29 nodeName:}" failed. No retries permitted until 2026-09-11 16:33:11.413602637 +0000 UTC m=+967.094707058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls") pod "isvc-secondary-d91b1e-predictor-5545848b98-gpftx" (UID: "c670fabd-99b5-48fc-a356-a88c9c460d29") : secret "isvc-secondary-d91b1e-predictor-serving-cert" not found
Sep 11 16:33:10.913895 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.913878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c670fabd-99b5-48fc-a356-a88c9c460d29-kserve-provision-location\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.914034 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.914015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.914094 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.914078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-cabundle-cert\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:10.921890 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:10.921870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvdp\" (UniqueName: \"kubernetes.io/projected/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-api-access-tlvdp\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:11.417643 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:11.417606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:11.420077 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:11.420042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls\") pod \"isvc-secondary-d91b1e-predictor-5545848b98-gpftx\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") " pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:11.617714 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:11.617675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:11.885078 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:11.885044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"]
Sep 11 16:33:11.888074 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:33:11.888045    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc670fabd_99b5_48fc_a356_a88c9c460d29.slice/crio-75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a WatchSource:0}: Error finding container 75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a: Status 404 returned error can't find the container with id 75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a
Sep 11 16:33:12.140240 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:12.140156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:12.359212 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:12.359178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" event={"ID":"c670fabd-99b5-48fc-a356-a88c9c460d29","Type":"ContainerStarted","Data":"1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6"}
Sep 11 16:33:12.359422 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:12.359245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" event={"ID":"c670fabd-99b5-48fc-a356-a88c9c460d29","Type":"ContainerStarted","Data":"75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a"}
Sep 11 16:33:16.131115 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:16.131067    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:33:16.131524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:16.131500    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:18.389234 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:18.389209    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/0.log"
Sep 11 16:33:18.389615 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:18.389245    2580 generic.go:356] "Generic (PLEG): container finished" podID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerID="1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6" exitCode=1
Sep 11 16:33:18.389615 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:18.389278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" event={"ID":"c670fabd-99b5-48fc-a356-a88c9c460d29","Type":"ContainerDied","Data":"1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6"}
Sep 11 16:33:19.395858 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:19.395812    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/0.log"
Sep 11 16:33:19.396257 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:19.395893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" event={"ID":"c670fabd-99b5-48fc-a356-a88c9c460d29","Type":"ContainerStarted","Data":"0fac30432eb898014136ae4c6339b5faf15bcbd9b6a4a48af31f0d026545e199"}
Sep 11 16:33:24.417498 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.417470    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/1.log"
Sep 11 16:33:24.417907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.417864    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/0.log"
Sep 11 16:33:24.417907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.417893    2580 generic.go:356] "Generic (PLEG): container finished" podID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerID="0fac30432eb898014136ae4c6339b5faf15bcbd9b6a4a48af31f0d026545e199" exitCode=1
Sep 11 16:33:24.417992 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.417928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" event={"ID":"c670fabd-99b5-48fc-a356-a88c9c460d29","Type":"ContainerDied","Data":"0fac30432eb898014136ae4c6339b5faf15bcbd9b6a4a48af31f0d026545e199"}
Sep 11 16:33:24.417992 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.417979    2580 scope.go:117] "RemoveContainer" containerID="1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6"
Sep 11 16:33:24.418346 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.418325    2580 scope.go:117] "RemoveContainer" containerID="1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6"
Sep 11 16:33:24.427715 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:33:24.427694    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d91b1e-predictor-5545848b98-gpftx_kserve-ci-e2e-test(c670fabd-99b5-48fc-a356-a88c9c460d29)\"" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29"
Sep 11 16:33:24.427958 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:33:24.427932    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_kserve-ci-e2e-test_c670fabd-99b5-48fc-a356-a88c9c460d29_0 in pod sandbox 75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a from index: no such id: '1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6'" containerID="1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6"
Sep 11 16:33:24.428015 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:24.427967    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_kserve-ci-e2e-test_c670fabd-99b5-48fc-a356-a88c9c460d29_0 in pod sandbox 75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a from index: no such id: '1f8910cdd98c0870ad3924fb02947ae9ec29492e6cb574cd2c1e6bf16468bee6'"
Sep 11 16:33:25.423123 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:25.423099    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/1.log"
Sep 11 16:33:26.131672 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:26.131633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:33:26.131855 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:26.131718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:33:28.755449 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:28.755372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"]
Sep 11 16:33:28.791154 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:28.791114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:28.852206 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:28.852177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:28.882492 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:28.882472    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/1.log"
Sep 11 16:33:28.884516 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:28.884501    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:29.065188 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065117    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-cabundle-cert\") pod \"c670fabd-99b5-48fc-a356-a88c9c460d29\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") "
Sep 11 16:33:29.065317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-rbac-proxy-sar-config\") pod \"c670fabd-99b5-48fc-a356-a88c9c460d29\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") "
Sep 11 16:33:29.065317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlvdp\" (UniqueName: \"kubernetes.io/projected/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-api-access-tlvdp\") pod \"c670fabd-99b5-48fc-a356-a88c9c460d29\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") "
Sep 11 16:33:29.065317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c670fabd-99b5-48fc-a356-a88c9c460d29-kserve-provision-location\") pod \"c670fabd-99b5-48fc-a356-a88c9c460d29\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") "
Sep 11 16:33:29.065434 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls\") pod \"c670fabd-99b5-48fc-a356-a88c9c460d29\" (UID: \"c670fabd-99b5-48fc-a356-a88c9c460d29\") "
Sep 11 16:33:29.065434 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c670fabd-99b5-48fc-a356-a88c9c460d29" (UID: "c670fabd-99b5-48fc-a356-a88c9c460d29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:29.065568 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c670fabd-99b5-48fc-a356-a88c9c460d29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c670fabd-99b5-48fc-a356-a88c9c460d29" (UID: "c670fabd-99b5-48fc-a356-a88c9c460d29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:29.065568 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c670fabd-99b5-48fc-a356-a88c9c460d29" (UID: "c670fabd-99b5-48fc-a356-a88c9c460d29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:29.065682 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.065598    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:29.067337 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.067313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c670fabd-99b5-48fc-a356-a88c9c460d29" (UID: "c670fabd-99b5-48fc-a356-a88c9c460d29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:29.067423 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.067388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-api-access-tlvdp" (OuterVolumeSpecName: "kube-api-access-tlvdp") pod "c670fabd-99b5-48fc-a356-a88c9c460d29" (UID: "c670fabd-99b5-48fc-a356-a88c9c460d29"). InnerVolumeSpecName "kube-api-access-tlvdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:29.166654 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.166627    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:29.166654 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.166652    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlvdp\" (UniqueName: \"kubernetes.io/projected/c670fabd-99b5-48fc-a356-a88c9c460d29-kube-api-access-tlvdp\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:29.166811 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.166663    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c670fabd-99b5-48fc-a356-a88c9c460d29-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:29.166811 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.166673    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c670fabd-99b5-48fc-a356-a88c9c460d29-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:29.441196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.441169    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d91b1e-predictor-5545848b98-gpftx_c670fabd-99b5-48fc-a356-a88c9c460d29/storage-initializer/1.log"
Sep 11 16:33:29.441359 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.441214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx" event={"ID":"c670fabd-99b5-48fc-a356-a88c9c460d29","Type":"ContainerDied","Data":"75769ba2cfebfb98622f1d3c94eb6a34fa46a68d095dc218d02cf10fcb9a991a"}
Sep 11 16:33:29.441359 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.441260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"
Sep 11 16:33:29.441359 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.441271    2580 scope.go:117] "RemoveContainer" containerID="0fac30432eb898014136ae4c6339b5faf15bcbd9b6a4a48af31f0d026545e199"
Sep 11 16:33:29.478742 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.478712    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"]
Sep 11 16:33:29.482577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:29.482556    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d91b1e-predictor-5545848b98-gpftx"]
Sep 11 16:33:30.942473 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:30.942431    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" path="/var/lib/kubelet/pods/c670fabd-99b5-48fc-a356-a88c9c460d29/volumes"
Sep 11 16:33:33.913556 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.913526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:33.957768 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.957741    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"]
Sep 11 16:33:33.958077 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.958066    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerName="storage-initializer"
Sep 11 16:33:33.958129 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.958079    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerName="storage-initializer"
Sep 11 16:33:33.958129 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.958102    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerName="storage-initializer"
Sep 11 16:33:33.958129 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.958108    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerName="storage-initializer"
Sep 11 16:33:33.958223 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.958160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerName="storage-initializer"
Sep 11 16:33:33.958223 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.958169    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c670fabd-99b5-48fc-a356-a88c9c460d29" containerName="storage-initializer"
Sep 11 16:33:33.960754 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.960738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:33.963441 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.963412    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-732eed\""
Sep 11 16:33:33.963554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.963411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-732eed-predictor-serving-cert\""
Sep 11 16:33:33.963554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.963504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-732eed-dockercfg-nrj92\""
Sep 11 16:33:33.963701 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.963681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-732eed-kube-rbac-proxy-sar-config\""
Sep 11 16:33:33.971542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:33.971521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"]
Sep 11 16:33:34.004462 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.004429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f81809-8510-470f-8af5-25e7c598f3ee-kserve-provision-location\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.004657 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.004634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.004730 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.004671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsr72\" (UniqueName: \"kubernetes.io/projected/93f81809-8510-470f-8af5-25e7c598f3ee-kube-api-access-nsr72\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.005022 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.004987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-cabundle-cert\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.005141 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.005085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f81809-8510-470f-8af5-25e7c598f3ee-proxy-tls\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.039643 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.039617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:34.106346 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.106317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-cabundle-cert\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.106445 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.106356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f81809-8510-470f-8af5-25e7c598f3ee-proxy-tls\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.106502 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.106482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f81809-8510-470f-8af5-25e7c598f3ee-kserve-provision-location\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.106583 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.106565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.106623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.106599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsr72\" (UniqueName: \"kubernetes.io/projected/93f81809-8510-470f-8af5-25e7c598f3ee-kube-api-access-nsr72\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.106838 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.106808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f81809-8510-470f-8af5-25e7c598f3ee-kserve-provision-location\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.107096 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.107081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-cabundle-cert\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.107199 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.107180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.108833 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.108801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f81809-8510-470f-8af5-25e7c598f3ee-proxy-tls\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.115510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.115491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsr72\" (UniqueName: \"kubernetes.io/projected/93f81809-8510-470f-8af5-25e7c598f3ee-kube-api-access-nsr72\") pod \"isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.271049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.270961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:34.539770 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:34.539741    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"]
Sep 11 16:33:34.541402 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:33:34.541376    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f81809_8510_470f_8af5_25e7c598f3ee.slice/crio-a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f WatchSource:0}: Error finding container a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f: Status 404 returned error can't find the container with id a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f
Sep 11 16:33:35.468025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:35.467986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" event={"ID":"93f81809-8510-470f-8af5-25e7c598f3ee","Type":"ContainerStarted","Data":"d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143"}
Sep 11 16:33:35.468392 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:35.468045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" event={"ID":"93f81809-8510-470f-8af5-25e7c598f3ee","Type":"ContainerStarted","Data":"a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f"}
Sep 11 16:33:40.490450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.490377    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/0.log"
Sep 11 16:33:40.490450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.490412    2580 generic.go:356] "Generic (PLEG): container finished" podID="93f81809-8510-470f-8af5-25e7c598f3ee" containerID="d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143" exitCode=1
Sep 11 16:33:40.490450 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.490439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" event={"ID":"93f81809-8510-470f-8af5-25e7c598f3ee","Type":"ContainerDied","Data":"d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143"}
Sep 11 16:33:40.849319 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.849243    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"]
Sep 11 16:33:40.849591 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.849561    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" containerID="cri-o://584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60" gracePeriod=30
Sep 11 16:33:40.849806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.849575    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" containerID="cri-o://42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac" gracePeriod=30
Sep 11 16:33:40.849940 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.849660    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" containerID="cri-o://3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391" gracePeriod=30
Sep 11 16:33:40.880309 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.880276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:40.947351 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:40.947318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:41.024382 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:41.024320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:41.236988 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:41.236915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:41.496685 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:41.496594    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerID="3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391" exitCode=2
Sep 11 16:33:41.496685 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:41.496666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerDied","Data":"3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391"}
Sep 11 16:33:41.498490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:41.498473    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/0.log"
Sep 11 16:33:41.498610 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:41.498524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" event={"ID":"93f81809-8510-470f-8af5-25e7c598f3ee","Type":"ContainerStarted","Data":"b8a6e70bd7c6d0683f20b42eba27659d3dcd5020ae11d64c0783fc3fb3349214"}
Sep 11 16:33:43.204903 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.204851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:43.510004 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.509980    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/1.log"
Sep 11 16:33:43.510408 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.510390    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/0.log"
Sep 11 16:33:43.510462 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.510425    2580 generic.go:356] "Generic (PLEG): container finished" podID="93f81809-8510-470f-8af5-25e7c598f3ee" containerID="b8a6e70bd7c6d0683f20b42eba27659d3dcd5020ae11d64c0783fc3fb3349214" exitCode=1
Sep 11 16:33:43.510515 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.510498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" event={"ID":"93f81809-8510-470f-8af5-25e7c598f3ee","Type":"ContainerDied","Data":"b8a6e70bd7c6d0683f20b42eba27659d3dcd5020ae11d64c0783fc3fb3349214"}
Sep 11 16:33:43.510571 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.510561    2580 scope.go:117] "RemoveContainer" containerID="d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143"
Sep 11 16:33:43.510935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.510917    2580 scope.go:117] "RemoveContainer" containerID="d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143"
Sep 11 16:33:43.527902 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:33:43.527882    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_kserve-ci-e2e-test(93f81809-8510-470f-8af5-25e7c598f3ee)\"" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" podUID="93f81809-8510-470f-8af5-25e7c598f3ee"
Sep 11 16:33:43.528040 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:33:43.528013    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_kserve-ci-e2e-test_93f81809-8510-470f-8af5-25e7c598f3ee_0 in pod sandbox a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f from index: no such id: 'd041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143'" containerID="d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143"
Sep 11 16:33:43.528099 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.528048    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_kserve-ci-e2e-test_93f81809-8510-470f-8af5-25e7c598f3ee_0 in pod sandbox a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f from index: no such id: 'd041070afb45b8e1507739d34d0463cf577da089b45fd0ffe1363212a8479143'"
Sep 11 16:33:43.944479 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.944446    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"]
Sep 11 16:33:43.965309 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:43.965279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:44.515815 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.515787    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/1.log"
Sep 11 16:33:44.643969 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.643946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/1.log"
Sep 11 16:33:44.645914 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.645896    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:44.695380 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f81809-8510-470f-8af5-25e7c598f3ee-kserve-provision-location\") pod \"93f81809-8510-470f-8af5-25e7c598f3ee\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") "
Sep 11 16:33:44.695547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-cabundle-cert\") pod \"93f81809-8510-470f-8af5-25e7c598f3ee\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") "
Sep 11 16:33:44.695547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-kube-rbac-proxy-sar-config\") pod \"93f81809-8510-470f-8af5-25e7c598f3ee\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") "
Sep 11 16:33:44.695547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsr72\" (UniqueName: \"kubernetes.io/projected/93f81809-8510-470f-8af5-25e7c598f3ee-kube-api-access-nsr72\") pod \"93f81809-8510-470f-8af5-25e7c598f3ee\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") "
Sep 11 16:33:44.695547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f81809-8510-470f-8af5-25e7c598f3ee-proxy-tls\") pod \"93f81809-8510-470f-8af5-25e7c598f3ee\" (UID: \"93f81809-8510-470f-8af5-25e7c598f3ee\") "
Sep 11 16:33:44.695736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f81809-8510-470f-8af5-25e7c598f3ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93f81809-8510-470f-8af5-25e7c598f3ee" (UID: "93f81809-8510-470f-8af5-25e7c598f3ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:44.695802 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695730    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93f81809-8510-470f-8af5-25e7c598f3ee" (UID: "93f81809-8510-470f-8af5-25e7c598f3ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:44.695802 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93f81809-8510-470f-8af5-25e7c598f3ee" (UID: "93f81809-8510-470f-8af5-25e7c598f3ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:44.695942 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695846    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f81809-8510-470f-8af5-25e7c598f3ee-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.695942 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695862    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.695942 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.695871    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f81809-8510-470f-8af5-25e7c598f3ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.697547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.697522    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f81809-8510-470f-8af5-25e7c598f3ee-kube-api-access-nsr72" (OuterVolumeSpecName: "kube-api-access-nsr72") pod "93f81809-8510-470f-8af5-25e7c598f3ee" (UID: "93f81809-8510-470f-8af5-25e7c598f3ee"). InnerVolumeSpecName "kube-api-access-nsr72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:44.697632 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.697525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f81809-8510-470f-8af5-25e7c598f3ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93f81809-8510-470f-8af5-25e7c598f3ee" (UID: "93f81809-8510-470f-8af5-25e7c598f3ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:44.796607 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.796514    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsr72\" (UniqueName: \"kubernetes.io/projected/93f81809-8510-470f-8af5-25e7c598f3ee-kube-api-access-nsr72\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.796607 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:44.796545    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f81809-8510-470f-8af5-25e7c598f3ee-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:33:45.127097 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.127059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 11 16:33:45.523468 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.523386    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerID="584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60" exitCode=0
Sep 11 16:33:45.523882 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.523457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerDied","Data":"584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60"}
Sep 11 16:33:45.524659 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.524639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx_93f81809-8510-470f-8af5-25e7c598f3ee/storage-initializer/1.log"
Sep 11 16:33:45.524749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.524732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"
Sep 11 16:33:45.524795 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.524761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx" event={"ID":"93f81809-8510-470f-8af5-25e7c598f3ee","Type":"ContainerDied","Data":"a50300a242f1a710d5a28a0ca3febe355c2e2345137f89a7c84cdeb7a27ed39f"}
Sep 11 16:33:45.524854 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.524841    2580 scope.go:117] "RemoveContainer" containerID="b8a6e70bd7c6d0683f20b42eba27659d3dcd5020ae11d64c0783fc3fb3349214"
Sep 11 16:33:45.557469 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.557436    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"]
Sep 11 16:33:45.561162 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:45.561136    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-732eed-predictor-795b6b9bcb-kprcx"]
Sep 11 16:33:46.131103 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:46.131053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:33:46.131358 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:46.131336    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:46.939718 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:46.939686    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" path="/var/lib/kubelet/pods/93f81809-8510-470f-8af5-25e7c598f3ee/volumes"
Sep 11 16:33:49.159765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:49.159726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:49.784570 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:49.784537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:50.126691 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:50.126650    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 11 16:33:55.126949 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:55.126909    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 11 16:33:55.127431 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:55.127031    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:33:56.131517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:56.131466    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:33:56.132016 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:33:56.131877    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:34:00.127057 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:00.127015    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 11 16:34:05.127044 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:05.127005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 11 16:34:06.131666 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:06.131617    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:34:06.132152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:06.131785    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:34:06.132152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:06.131890    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:34:06.132152 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:06.131976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:34:10.126878 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:10.126814    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 11 16:34:11.495773 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.495749    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:34:11.636070 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.635971    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-cabundle-cert\") pod \"7e180306-c3db-4197-9a4c-7810ee0d7434\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") "
Sep 11 16:34:11.636070 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e180306-c3db-4197-9a4c-7810ee0d7434-kserve-provision-location\") pod \"7e180306-c3db-4197-9a4c-7810ee0d7434\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") "
Sep 11 16:34:11.636320 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pmls\" (UniqueName: \"kubernetes.io/projected/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-api-access-7pmls\") pod \"7e180306-c3db-4197-9a4c-7810ee0d7434\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") "
Sep 11 16:34:11.636320 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e180306-c3db-4197-9a4c-7810ee0d7434-proxy-tls\") pod \"7e180306-c3db-4197-9a4c-7810ee0d7434\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") "
Sep 11 16:34:11.636320 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-rbac-proxy-sar-config\") pod \"7e180306-c3db-4197-9a4c-7810ee0d7434\" (UID: \"7e180306-c3db-4197-9a4c-7810ee0d7434\") "
Sep 11 16:34:11.636484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e180306-c3db-4197-9a4c-7810ee0d7434-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e180306-c3db-4197-9a4c-7810ee0d7434" (UID: "7e180306-c3db-4197-9a4c-7810ee0d7434"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:34:11.636484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e180306-c3db-4197-9a4c-7810ee0d7434" (UID: "7e180306-c3db-4197-9a4c-7810ee0d7434"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:34:11.636647 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.636615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e180306-c3db-4197-9a4c-7810ee0d7434" (UID: "7e180306-c3db-4197-9a4c-7810ee0d7434"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:34:11.638193 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.638170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-api-access-7pmls" (OuterVolumeSpecName: "kube-api-access-7pmls") pod "7e180306-c3db-4197-9a4c-7810ee0d7434" (UID: "7e180306-c3db-4197-9a4c-7810ee0d7434"). InnerVolumeSpecName "kube-api-access-7pmls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:34:11.638349 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.638334    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e180306-c3db-4197-9a4c-7810ee0d7434-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e180306-c3db-4197-9a4c-7810ee0d7434" (UID: "7e180306-c3db-4197-9a4c-7810ee0d7434"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:34:11.642269 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.642246    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerID="42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac" exitCode=0
Sep 11 16:34:11.642384 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.642307    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"
Sep 11 16:34:11.642384 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.642333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerDied","Data":"42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac"}
Sep 11 16:34:11.642465 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.642415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm" event={"ID":"7e180306-c3db-4197-9a4c-7810ee0d7434","Type":"ContainerDied","Data":"c6a896e9a6183695cc73f4f33bf755a836a1eb9f6a9f36848e7be88564c2c69c"}
Sep 11 16:34:11.642501 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.642465    2580 scope.go:117] "RemoveContainer" containerID="42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac"
Sep 11 16:34:11.654609 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.654587    2580 scope.go:117] "RemoveContainer" containerID="3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391"
Sep 11 16:34:11.662512 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.662495    2580 scope.go:117] "RemoveContainer" containerID="584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60"
Sep 11 16:34:11.667416 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.667391    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"]
Sep 11 16:34:11.670623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.670601    2580 scope.go:117] "RemoveContainer" containerID="387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e"
Sep 11 16:34:11.671283 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.671264    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ecc5e-predictor-76bd699f58-xdhqm"]
Sep 11 16:34:11.677968 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.677948    2580 scope.go:117] "RemoveContainer" containerID="42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac"
Sep 11 16:34:11.678231 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:34:11.678210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac\": container with ID starting with 42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac not found: ID does not exist" containerID="42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac"
Sep 11 16:34:11.678274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.678240    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac"} err="failed to get container status \"42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac\": rpc error: code = NotFound desc = could not find container \"42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac\": container with ID starting with 42f595fbd4547ca2695d9f2eabcd4ebda019375e8f3b990186b4c6a6e899b4ac not found: ID does not exist"
Sep 11 16:34:11.678274 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.678259    2580 scope.go:117] "RemoveContainer" containerID="3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391"
Sep 11 16:34:11.678498 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:34:11.678478    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391\": container with ID starting with 3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391 not found: ID does not exist" containerID="3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391"
Sep 11 16:34:11.678546 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.678505    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391"} err="failed to get container status \"3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391\": rpc error: code = NotFound desc = could not find container \"3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391\": container with ID starting with 3e9dcab52114e19bbcd5f1f8a5a72dbf25d0c97d758f764353cd241e6e6dd391 not found: ID does not exist"
Sep 11 16:34:11.678546 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.678520    2580 scope.go:117] "RemoveContainer" containerID="584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60"
Sep 11 16:34:11.678751 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:34:11.678735    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60\": container with ID starting with 584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60 not found: ID does not exist" containerID="584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60"
Sep 11 16:34:11.678796 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.678755    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60"} err="failed to get container status \"584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60\": rpc error: code = NotFound desc = could not find container \"584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60\": container with ID starting with 584128720ff4d3fefe0c016c3803a9ff0dd2b153915a9e5ff4fb48674e836b60 not found: ID does not exist"
Sep 11 16:34:11.678796 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.678769    2580 scope.go:117] "RemoveContainer" containerID="387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e"
Sep 11 16:34:11.679011 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:34:11.678994    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e\": container with ID starting with 387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e not found: ID does not exist" containerID="387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e"
Sep 11 16:34:11.679063 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.679015    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e"} err="failed to get container status \"387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e\": rpc error: code = NotFound desc = could not find container \"387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e\": container with ID starting with 387c41016b1aaf762cf9fca0b9fc28eb42f133c2cf17f5c2cc39cfc87df8d72e not found: ID does not exist"
Sep 11 16:34:11.737530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.737479    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:34:11.737530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.737527    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e180306-c3db-4197-9a4c-7810ee0d7434-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:34:11.737530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.737538    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmls\" (UniqueName: \"kubernetes.io/projected/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-api-access-7pmls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:34:11.737530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.737547    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e180306-c3db-4197-9a4c-7810ee0d7434-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:34:11.737774 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:11.737558    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e180306-c3db-4197-9a4c-7810ee0d7434-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:34:12.940155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:12.940122    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" path="/var/lib/kubelet/pods/7e180306-c3db-4197-9a4c-7810ee0d7434/volumes"
Sep 11 16:34:53.679692 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:53.679649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:53.744238 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:34:53.744207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:02.168377 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:02.168346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:02.225446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:02.225414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:29.348202 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:29.348168    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:34.453384 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.453346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:34.508419 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508382    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"]
Sep 11 16:35:34.508729 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508714    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="storage-initializer"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508730    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="storage-initializer"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508741    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508747    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508757    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" containerName="storage-initializer"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508762    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" containerName="storage-initializer"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508780    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" containerName="storage-initializer"
Sep 11 16:35:34.508783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508785    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" containerName="storage-initializer"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508792    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508806    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508811    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508868    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kube-rbac-proxy"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508878    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" containerName="storage-initializer"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508886    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="kserve-container"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508892    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f81809-8510-470f-8af5-25e7c598f3ee" containerName="storage-initializer"
Sep 11 16:35:34.509025 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.508899    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e180306-c3db-4197-9a4c-7810ee0d7434" containerName="agent"
Sep 11 16:35:34.511979 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.511960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.514930 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.514905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:35:34.515058 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.515039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:35:34.515109 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.515049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:35:34.515109 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.515049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-06028-kube-rbac-proxy-sar-config\""
Sep 11 16:35:34.515195 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.515130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:35:34.516058 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.516041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-06028-predictor-serving-cert\""
Sep 11 16:35:34.516147 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.516112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:35:34.521636 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.521614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"]
Sep 11 16:35:34.705393 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.705305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-cabundle-cert\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.705393 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.705364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.705584 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.705439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kserve-provision-location\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.705584 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.705519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhm8\" (UniqueName: \"kubernetes.io/projected/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-api-access-rkhm8\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.705584 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.705565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.806433 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.806399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhm8\" (UniqueName: \"kubernetes.io/projected/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-api-access-rkhm8\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.806600 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.806444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.806600 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.806474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-cabundle-cert\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.806600 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.806507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.806600 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.806534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kserve-provision-location\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.806817 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:35:34.806701    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-serving-cert: secret "raw-sklearn-runtime-06028-predictor-serving-cert" not found
Sep 11 16:35:34.806817 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:35:34.806793    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls podName:e636b8c3-2419-4d02-bb0f-caa1f285dbdc nodeName:}" failed. No retries permitted until 2026-09-11 16:35:35.306770718 +0000 UTC m=+1110.987875142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls") pod "raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc") : secret "raw-sklearn-runtime-06028-predictor-serving-cert" not found
Sep 11 16:35:34.807011 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.806986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kserve-provision-location\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.807196 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.807173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.807239 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.807201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-cabundle-cert\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:34.815427 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:34.815405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhm8\" (UniqueName: \"kubernetes.io/projected/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-api-access-rkhm8\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:35.311641 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:35.311600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:35.311854 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:35:35.311763    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-serving-cert: secret "raw-sklearn-runtime-06028-predictor-serving-cert" not found
Sep 11 16:35:35.311903 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:35:35.311861    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls podName:e636b8c3-2419-4d02-bb0f-caa1f285dbdc nodeName:}" failed. No retries permitted until 2026-09-11 16:35:36.311844098 +0000 UTC m=+1111.992948518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls") pod "raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc") : secret "raw-sklearn-runtime-06028-predictor-serving-cert" not found
Sep 11 16:35:35.508294 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:35.508257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:36.319950 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:36.319910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:36.322402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:36.322375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") pod \"raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:36.322554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:36.322520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:36.594129 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:36.594103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"]
Sep 11 16:35:36.595836 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:35:36.595802    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode636b8c3_2419_4d02_bb0f_caa1f285dbdc.slice/crio-f3cce20dfde0bf2d96533beed59b99496a5421bf907febbc89dc3f6ef5e0ac02 WatchSource:0}: Error finding container f3cce20dfde0bf2d96533beed59b99496a5421bf907febbc89dc3f6ef5e0ac02: Status 404 returned error can't find the container with id f3cce20dfde0bf2d96533beed59b99496a5421bf907febbc89dc3f6ef5e0ac02
Sep 11 16:35:36.986224 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:36.986187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerStarted","Data":"626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29"}
Sep 11 16:35:36.986393 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:36.986242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerStarted","Data":"f3cce20dfde0bf2d96533beed59b99496a5421bf907febbc89dc3f6ef5e0ac02"}
Sep 11 16:35:41.003567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:41.003531    2580 generic.go:356] "Generic (PLEG): container finished" podID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerID="626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29" exitCode=0
Sep 11 16:35:41.003970 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:41.003584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerDied","Data":"626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29"}
Sep 11 16:35:42.009530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:42.009498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerStarted","Data":"3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51"}
Sep 11 16:35:42.009961 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:42.009556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerStarted","Data":"d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d"}
Sep 11 16:35:42.009961 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:42.009747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:42.031276 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:42.031228    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podStartSLOduration=4.248824997 podStartE2EDuration="8.03121335s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782388353s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:35:42.029915312 +0000 UTC m=+1117.711019738" watchObservedRunningTime="2026-09-11 16:35:42.03121335 +0000 UTC m=+1117.712317839"
Sep 11 16:35:43.013713 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:43.013679    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:43.014908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:43.014869    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:35:44.017116 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:44.017064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:35:49.021761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:49.021730    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:35:49.022212 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:49.022184    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:35:51.509517 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:51.509479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:59.022174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:35:59.022133    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:36:09.022576 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:09.022536    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:36:19.022286 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:19.022240    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:36:29.022815 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:29.022729    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:36:39.022602 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:39.022563    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:36:41.102280 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:41.102248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:46.248323 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.248263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:46.335502 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.335466    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"]
Sep 11 16:36:46.338156 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.338131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:46.339137 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.339120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.341778 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.341758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 11 16:36:46.341894 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.341785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 11 16:36:46.348999 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.348980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"]
Sep 11 16:36:46.418776 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.418743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.418948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.418779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d9w\" (UniqueName: \"kubernetes.io/projected/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-api-access-s7d9w\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.418948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.418805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.418948 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.418889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.419060 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.418980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.444865 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.444838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:46.520039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.519954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520225 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520225 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520225 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d9w\" (UniqueName: \"kubernetes.io/projected/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-api-access-s7d9w\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520373 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520774 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.520912 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.520786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.522587 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.522556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.528542 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.528522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d9w\" (UniqueName: \"kubernetes.io/projected/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-api-access-s7d9w\") pod \"isvc-canary-create-predictor-6474b875b9-zgslm\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.546084 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.546060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:46.649566 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.649544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:46.921287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:46.921262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"]
Sep 11 16:36:46.923265 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:36:46.923236    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ed18ed_9b8b_4de8_80fc_d64d9243df11.slice/crio-6a3be3aecf4498ca866189115edee26f609f4b544429f7799492385596b05b51 WatchSource:0}: Error finding container 6a3be3aecf4498ca866189115edee26f609f4b544429f7799492385596b05b51: Status 404 returned error can't find the container with id 6a3be3aecf4498ca866189115edee26f609f4b544429f7799492385596b05b51
Sep 11 16:36:47.266907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:47.266808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerStarted","Data":"e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04"}
Sep 11 16:36:47.266907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:47.266893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerStarted","Data":"6a3be3aecf4498ca866189115edee26f609f4b544429f7799492385596b05b51"}
Sep 11 16:36:49.022473 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:49.022431    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:36:51.286573 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:51.286488    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerID="e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04" exitCode=0
Sep 11 16:36:51.287041 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:51.286571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerDied","Data":"e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04"}
Sep 11 16:36:52.293040 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:52.293005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerStarted","Data":"911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83"}
Sep 11 16:36:52.293386 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:52.293069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerStarted","Data":"d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778"}
Sep 11 16:36:52.293386 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:52.293259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:52.313885 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:52.313839    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podStartSLOduration=2.431892909 podStartE2EDuration="6.313810191s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.881917282s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:52.313084977 +0000 UTC m=+1187.994189402" watchObservedRunningTime="2026-09-11 16:36:52.313810191 +0000 UTC m=+1187.994914615"
Sep 11 16:36:53.296880 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:53.296846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:53.297908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:53.297884    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:36:54.300378 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:54.300332    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:36:59.022931 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:59.022901    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:36:59.305194 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:59.305106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:36:59.305707 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:36:59.305679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:37:04.592161 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:04.592126    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"]
Sep 11 16:37:04.592549 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:04.592370    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" containerID="cri-o://d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d" gracePeriod=30
Sep 11 16:37:04.592549 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:04.592413    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kube-rbac-proxy" containerID="cri-o://3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51" gracePeriod=30
Sep 11 16:37:04.616603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:04.616571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:05.348304 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:05.348272    2580 generic.go:356] "Generic (PLEG): container finished" podID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerID="3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51" exitCode=2
Sep 11 16:37:05.348486 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:05.348326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerDied","Data":"3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51"}
Sep 11 16:37:09.018090 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.018048    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 11 16:37:09.022424 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.022393    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:37:09.235793 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.235766    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:37:09.306724 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.306650    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:37:09.318381 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkhm8\" (UniqueName: \"kubernetes.io/projected/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-api-access-rkhm8\") pod \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") "
Sep 11 16:37:09.318477 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kserve-provision-location\") pod \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") "
Sep 11 16:37:09.318477 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-cabundle-cert\") pod \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") "
Sep 11 16:37:09.318567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") pod \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") "
Sep 11 16:37:09.318567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-rbac-proxy-sar-config\") pod \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\" (UID: \"e636b8c3-2419-4d02-bb0f-caa1f285dbdc\") "
Sep 11 16:37:09.318840 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e636b8c3-2419-4d02-bb0f-caa1f285dbdc" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:09.318932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e636b8c3-2419-4d02-bb0f-caa1f285dbdc" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:37:09.318932 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.318908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e636b8c3-2419-4d02-bb0f-caa1f285dbdc" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:37:09.320412 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.320379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-api-access-rkhm8" (OuterVolumeSpecName: "kube-api-access-rkhm8") pod "e636b8c3-2419-4d02-bb0f-caa1f285dbdc" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc"). InnerVolumeSpecName "kube-api-access-rkhm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:37:09.320510 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.320432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e636b8c3-2419-4d02-bb0f-caa1f285dbdc" (UID: "e636b8c3-2419-4d02-bb0f-caa1f285dbdc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:37:09.366687 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.366663    2580 generic.go:356] "Generic (PLEG): container finished" podID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerID="d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d" exitCode=0
Sep 11 16:37:09.366790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.366723    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"
Sep 11 16:37:09.366790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.366735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerDied","Data":"d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d"}
Sep 11 16:37:09.366790 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.366783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f" event={"ID":"e636b8c3-2419-4d02-bb0f-caa1f285dbdc","Type":"ContainerDied","Data":"f3cce20dfde0bf2d96533beed59b99496a5421bf907febbc89dc3f6ef5e0ac02"}
Sep 11 16:37:09.366921 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.366808    2580 scope.go:117] "RemoveContainer" containerID="3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51"
Sep 11 16:37:09.374991 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.374974    2580 scope.go:117] "RemoveContainer" containerID="d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d"
Sep 11 16:37:09.381978 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.381960    2580 scope.go:117] "RemoveContainer" containerID="626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29"
Sep 11 16:37:09.389668 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.389652    2580 scope.go:117] "RemoveContainer" containerID="3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51"
Sep 11 16:37:09.389988 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.389970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"]
Sep 11 16:37:09.390055 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:37:09.389970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51\": container with ID starting with 3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51 not found: ID does not exist" containerID="3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51"
Sep 11 16:37:09.390055 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.390018    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51"} err="failed to get container status \"3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51\": rpc error: code = NotFound desc = could not find container \"3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51\": container with ID starting with 3fd4c253139dd92f10371ff85d7cc56465e9b07d89d626bf6f9ac9b872431f51 not found: ID does not exist"
Sep 11 16:37:09.390055 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.390034    2580 scope.go:117] "RemoveContainer" containerID="d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d"
Sep 11 16:37:09.390272 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:37:09.390256    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d\": container with ID starting with d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d not found: ID does not exist" containerID="d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d"
Sep 11 16:37:09.390321 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.390276    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d"} err="failed to get container status \"d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d\": rpc error: code = NotFound desc = could not find container \"d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d\": container with ID starting with d733e0a24bee15b1a6f2e450493f5b226bf6723ef6c53afa6d7131db0f7a637d not found: ID does not exist"
Sep 11 16:37:09.390321 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.390300    2580 scope.go:117] "RemoveContainer" containerID="626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29"
Sep 11 16:37:09.390563 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:37:09.390548    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29\": container with ID starting with 626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29 not found: ID does not exist" containerID="626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29"
Sep 11 16:37:09.390606 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.390576    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29"} err="failed to get container status \"626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29\": rpc error: code = NotFound desc = could not find container \"626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29\": container with ID starting with 626401231f7c09913296768da0e3a113ac8071ee50110fb9148e9792e02b6e29 not found: ID does not exist"
Sep 11 16:37:09.393672 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.393654    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-06028-predictor-55c68d8cf6-58v4f"]
Sep 11 16:37:09.419685 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.419668    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:37:09.419754 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.419688    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:37:09.419754 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.419698    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:37:09.419754 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.419706    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:37:09.419754 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:09.419714    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhm8\" (UniqueName: \"kubernetes.io/projected/e636b8c3-2419-4d02-bb0f-caa1f285dbdc-kube-api-access-rkhm8\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:37:10.939346 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:10.939313    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" path="/var/lib/kubelet/pods/e636b8c3-2419-4d02-bb0f-caa1f285dbdc/volumes"
Sep 11 16:37:19.306155 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:19.306111    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:37:29.306402 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:29.306366    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:37:39.305654 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:39.305615    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:37:49.305915 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:49.305872    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:37:53.409536 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:53.409502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:59.306178 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:37:59.306097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:38:09.306641 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:09.306612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:38:13.815146 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:13.815106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:16.372668 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.372632    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"]
Sep 11 16:38:16.373085 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.372982    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" containerID="cri-o://d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778" gracePeriod=30
Sep 11 16:38:16.373085 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.373011    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kube-rbac-proxy" containerID="cri-o://911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83" gracePeriod=30
Sep 11 16:38:16.393734 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.393706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:16.449363 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.449330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:16.654735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.654647    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerID="911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83" exitCode=2
Sep 11 16:38:16.654735 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:16.654715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerDied","Data":"911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83"}
Sep 11 16:38:19.300765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:19.300715    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 11 16:38:19.306143 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:19.306112    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:38:21.221546 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.221517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:38:21.311783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.311692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kserve-provision-location\") pod \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") "
Sep 11 16:38:21.311783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.311762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7d9w\" (UniqueName: \"kubernetes.io/projected/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-api-access-s7d9w\") pod \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") "
Sep 11 16:38:21.311783 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.311779    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-rbac-proxy-sar-config\") pod \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") "
Sep 11 16:38:21.312128 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.311807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-proxy-tls\") pod \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") "
Sep 11 16:38:21.312128 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.311846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-cabundle-cert\") pod \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\" (UID: \"e7ed18ed-9b8b-4de8-80fc-d64d9243df11\") "
Sep 11 16:38:21.312238 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.312170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7ed18ed-9b8b-4de8-80fc-d64d9243df11" (UID: "e7ed18ed-9b8b-4de8-80fc-d64d9243df11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:21.312238 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.312208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7ed18ed-9b8b-4de8-80fc-d64d9243df11" (UID: "e7ed18ed-9b8b-4de8-80fc-d64d9243df11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:21.312314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.312239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7ed18ed-9b8b-4de8-80fc-d64d9243df11" (UID: "e7ed18ed-9b8b-4de8-80fc-d64d9243df11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:21.313984 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.313958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-api-access-s7d9w" (OuterVolumeSpecName: "kube-api-access-s7d9w") pod "e7ed18ed-9b8b-4de8-80fc-d64d9243df11" (UID: "e7ed18ed-9b8b-4de8-80fc-d64d9243df11"). InnerVolumeSpecName "kube-api-access-s7d9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:21.314068 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.313978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7ed18ed-9b8b-4de8-80fc-d64d9243df11" (UID: "e7ed18ed-9b8b-4de8-80fc-d64d9243df11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:21.413049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.413012    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.413049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.413045    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7d9w\" (UniqueName: \"kubernetes.io/projected/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-api-access-s7d9w\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.413049 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.413056    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.413282 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.413067    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.413282 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.413083    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed18ed-9b8b-4de8-80fc-d64d9243df11-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.534985 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.534948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:21.554308 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554276    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"]
Sep 11 16:38:21.554596 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554584    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="storage-initializer"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554598    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="storage-initializer"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554605    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kube-rbac-proxy"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554611    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kube-rbac-proxy"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554619    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554625    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554635    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kube-rbac-proxy"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554640    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kube-rbac-proxy"
Sep 11 16:38:21.554648 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554649    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554654    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554664    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="storage-initializer"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554669    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="storage-initializer"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554723    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kserve-container"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554734    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kserve-container"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554740    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerName="kube-rbac-proxy"
Sep 11 16:38:21.554959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.554746    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e636b8c3-2419-4d02-bb0f-caa1f285dbdc" containerName="kube-rbac-proxy"
Sep 11 16:38:21.557978 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.557961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.560624 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.560601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 11 16:38:21.560748 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.560667    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 11 16:38:21.566736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.566710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"]
Sep 11 16:38:21.607937 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.607900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:21.611655 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.611630    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"]
Sep 11 16:38:21.614697 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.614674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5be48400-8ded-4271-90a0-9d7901ad77ae-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.614808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.614718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9cx\" (UniqueName: \"kubernetes.io/projected/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-api-access-pm9cx\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.614808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.614741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.614959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.614841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.614959 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.614878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.615084 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.615069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.617943 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.617922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 11 16:38:21.626112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.626088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"]
Sep 11 16:38:21.676622 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.676586    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" containerID="d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778" exitCode=0
Sep 11 16:38:21.676808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.676657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerDied","Data":"d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778"}
Sep 11 16:38:21.676808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.676701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm" event={"ID":"e7ed18ed-9b8b-4de8-80fc-d64d9243df11","Type":"ContainerDied","Data":"6a3be3aecf4498ca866189115edee26f609f4b544429f7799492385596b05b51"}
Sep 11 16:38:21.676808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.676728    2580 scope.go:117] "RemoveContainer" containerID="911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83"
Sep 11 16:38:21.676808 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.676659    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"
Sep 11 16:38:21.686013 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.685992    2580 scope.go:117] "RemoveContainer" containerID="d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778"
Sep 11 16:38:21.694342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.694320    2580 scope.go:117] "RemoveContainer" containerID="e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04"
Sep 11 16:38:21.700218 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.700190    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"]
Sep 11 16:38:21.704789 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.704761    2580 scope.go:117] "RemoveContainer" containerID="911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83"
Sep 11 16:38:21.705261 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.705156    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83\": container with ID starting with 911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83 not found: ID does not exist" containerID="911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83"
Sep 11 16:38:21.705261 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.705192    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83"} err="failed to get container status \"911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83\": rpc error: code = NotFound desc = could not find container \"911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83\": container with ID starting with 911403a1ae052b4a0555a3f444b3e10982dda4640ae348cebd5b3e42a641bb83 not found: ID does not exist"
Sep 11 16:38:21.705261 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.705215    2580 scope.go:117] "RemoveContainer" containerID="d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778"
Sep 11 16:38:21.705566 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.705547    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778\": container with ID starting with d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778 not found: ID does not exist" containerID="d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778"
Sep 11 16:38:21.705620 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.705571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778"} err="failed to get container status \"d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778\": rpc error: code = NotFound desc = could not find container \"d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778\": container with ID starting with d09dfb4472ecb8fa923537374df5a8db8a81e3119767fc537f19de04d3c1d778 not found: ID does not exist"
Sep 11 16:38:21.705620 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.705591    2580 scope.go:117] "RemoveContainer" containerID="e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04"
Sep 11 16:38:21.705886 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.705866    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04\": container with ID starting with e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04 not found: ID does not exist" containerID="e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04"
Sep 11 16:38:21.705937 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.705896    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04"} err="failed to get container status \"e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04\": rpc error: code = NotFound desc = could not find container \"e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04\": container with ID starting with e2b294d4091188ac837235029828db660503160e3e820037d3f580290987bf04 not found: ID does not exist"
Sep 11 16:38:21.706955 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.706936    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zgslm"]
Sep 11 16:38:21.715446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.715528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.715528 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.715631 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5be48400-8ded-4271-90a0-9d7901ad77ae-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.715686 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbff78d9-8e06-4b43-b17a-ec37334979bc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.715743 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9cx\" (UniqueName: \"kubernetes.io/projected/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-api-access-pm9cx\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.715743 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.715907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.715907 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gt28\" (UniqueName: \"kubernetes.io/projected/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-api-access-5gt28\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.716019 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.716019 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.715976    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 11 16:38:21.716019 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.715975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5be48400-8ded-4271-90a0-9d7901ad77ae-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.716181 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.716038    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls podName:5be48400-8ded-4271-90a0-9d7901ad77ae nodeName:}" failed. No retries permitted until 2026-09-11 16:38:22.216023001 +0000 UTC m=+1277.897127404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls") pod "isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" (UID: "5be48400-8ded-4271-90a0-9d7901ad77ae") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 11 16:38:21.716251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.716222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.716312 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.716285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.724243 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.724215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9cx\" (UniqueName: \"kubernetes.io/projected/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-api-access-pm9cx\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:21.816512 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.816412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.816512 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.816458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gt28\" (UniqueName: \"kubernetes.io/projected/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-api-access-5gt28\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.816761 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.816570    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 11 16:38:21.816761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.816587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.816761 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:38:21.816641    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls podName:dbff78d9-8e06-4b43-b17a-ec37334979bc nodeName:}" failed. No retries permitted until 2026-09-11 16:38:22.316625272 +0000 UTC m=+1277.997729675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls") pod "isvc-canary-promote-predictor-7cf78f884f-jdmdp" (UID: "dbff78d9-8e06-4b43-b17a-ec37334979bc") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 11 16:38:21.816761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.816723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.816989 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.816818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbff78d9-8e06-4b43-b17a-ec37334979bc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.817172 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.817150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbff78d9-8e06-4b43-b17a-ec37334979bc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.817240 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.817222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.817277 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.817245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.825092 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.825073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gt28\" (UniqueName: \"kubernetes.io/projected/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-api-access-5gt28\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:21.894426 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.894378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:21.997237 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:21.997205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:22.220187 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.220149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:22.222484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.222454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-22x6h\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:22.321623 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.321578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:22.323989 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.323968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jdmdp\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:22.469687 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.469647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:22.525163 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.525129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:22.740168 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.740142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"]
Sep 11 16:38:22.742214 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:38:22.742189    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be48400_8ded_4271_90a0_9d7901ad77ae.slice/crio-764f6596d5e7a95f658eefc8e7c65248e817645da65f6af369c7b387f5b8b06e WatchSource:0}: Error finding container 764f6596d5e7a95f658eefc8e7c65248e817645da65f6af369c7b387f5b8b06e: Status 404 returned error can't find the container with id 764f6596d5e7a95f658eefc8e7c65248e817645da65f6af369c7b387f5b8b06e
Sep 11 16:38:22.802780 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.802753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"]
Sep 11 16:38:22.804726 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:38:22.804694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbff78d9_8e06_4b43_b17a_ec37334979bc.slice/crio-d773c97a9aff258f580e33158a2b8ad66bb38d2b124939038cabe386378dc71e WatchSource:0}: Error finding container d773c97a9aff258f580e33158a2b8ad66bb38d2b124939038cabe386378dc71e: Status 404 returned error can't find the container with id d773c97a9aff258f580e33158a2b8ad66bb38d2b124939038cabe386378dc71e
Sep 11 16:38:22.940663 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:22.940633    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ed18ed-9b8b-4de8-80fc-d64d9243df11" path="/var/lib/kubelet/pods/e7ed18ed-9b8b-4de8-80fc-d64d9243df11/volumes"
Sep 11 16:38:23.687287 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:23.687249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerStarted","Data":"5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5"}
Sep 11 16:38:23.687765 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:23.687346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerStarted","Data":"d773c97a9aff258f580e33158a2b8ad66bb38d2b124939038cabe386378dc71e"}
Sep 11 16:38:23.688794 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:23.688765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerStarted","Data":"57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294"}
Sep 11 16:38:23.688919 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:23.688847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerStarted","Data":"764f6596d5e7a95f658eefc8e7c65248e817645da65f6af369c7b387f5b8b06e"}
Sep 11 16:38:26.703002 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:26.702969    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerID="5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5" exitCode=0
Sep 11 16:38:26.703457 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:26.703046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerDied","Data":"5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5"}
Sep 11 16:38:26.704631 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:26.704611    2580 generic.go:356] "Generic (PLEG): container finished" podID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerID="57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294" exitCode=0
Sep 11 16:38:26.704717 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:26.704651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerDied","Data":"57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294"}
Sep 11 16:38:27.711330 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.711295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerStarted","Data":"9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106"}
Sep 11 16:38:27.711727 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.711352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerStarted","Data":"8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d"}
Sep 11 16:38:27.711727 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.711606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:27.713440 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.713417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerStarted","Data":"c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02"}
Sep 11 16:38:27.713550 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.713463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerStarted","Data":"1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935"}
Sep 11 16:38:27.713757 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.713741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:27.713834 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.713795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:27.715127 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.715104    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:38:27.729692 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.729647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podStartSLOduration=2.91445491 podStartE2EDuration="6.729633178s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.815178268s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:38:27.728809595 +0000 UTC m=+1283.409914043" watchObservedRunningTime="2026-09-11 16:38:27.729633178 +0000 UTC m=+1283.410737602"
Sep 11 16:38:27.748404 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:27.748361    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podStartSLOduration=2.8917883399999997 podStartE2EDuration="6.748348444s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.856560104s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:38:27.74787714 +0000 UTC m=+1283.428981568" watchObservedRunningTime="2026-09-11 16:38:27.748348444 +0000 UTC m=+1283.429452869"
Sep 11 16:38:28.717674 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:28.717637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:28.718150 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:28.717685    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:38:28.718918 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:28.718895    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:38:29.722089 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:29.722049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:38:33.722297 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:33.722265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:38:33.723064 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:33.723030    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:38:34.726280 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:34.726246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:38:34.726743 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:34.726715    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:38:43.723247 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:43.723202    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:38:44.726868 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:44.726810    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:38:53.723035 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:53.722994    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:38:54.727112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:38:54.727068    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:39:03.722950 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:03.722915    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:39:04.726806 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:04.726767    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:39:13.723237 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:13.723201    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:39:14.726944 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:14.726907    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:39:23.723433 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:23.723388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:39:24.727349 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:24.727304    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:39:33.723394 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:33.723303    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:39:34.727024 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:34.726980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:39:43.723988 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:43.723955    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:39:44.727535 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:44.727501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:39:51.592703 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:51.592671    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"]
Sep 11 16:39:51.593134 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:51.592921    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" containerID="cri-o://8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d" gracePeriod=30
Sep 11 16:39:51.593134 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:51.592946    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kube-rbac-proxy" containerID="cri-o://9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106" gracePeriod=30
Sep 11 16:39:51.619899 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:51.619849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:52.085023 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:52.084985    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerID="9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106" exitCode=2
Sep 11 16:39:52.085216 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:52.085054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerDied","Data":"9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106"}
Sep 11 16:39:54.723008 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:54.722965    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:39:54.727202 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:54.727173    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:39:55.947985 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.947952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:39:55.962308 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-rbac-proxy-sar-config\") pod \"dbff78d9-8e06-4b43-b17a-ec37334979bc\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") "
Sep 11 16:39:55.962439 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962319    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbff78d9-8e06-4b43-b17a-ec37334979bc-kserve-provision-location\") pod \"dbff78d9-8e06-4b43-b17a-ec37334979bc\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") "
Sep 11 16:39:55.962439 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gt28\" (UniqueName: \"kubernetes.io/projected/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-api-access-5gt28\") pod \"dbff78d9-8e06-4b43-b17a-ec37334979bc\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") "
Sep 11 16:39:55.962439 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-cabundle-cert\") pod \"dbff78d9-8e06-4b43-b17a-ec37334979bc\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") "
Sep 11 16:39:55.962617 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls\") pod \"dbff78d9-8e06-4b43-b17a-ec37334979bc\" (UID: \"dbff78d9-8e06-4b43-b17a-ec37334979bc\") "
Sep 11 16:39:55.962671 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbff78d9-8e06-4b43-b17a-ec37334979bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbff78d9-8e06-4b43-b17a-ec37334979bc" (UID: "dbff78d9-8e06-4b43-b17a-ec37334979bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:55.962731 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbff78d9-8e06-4b43-b17a-ec37334979bc" (UID: "dbff78d9-8e06-4b43-b17a-ec37334979bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:39:55.962792 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962732    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:39:55.962792 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962750    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbff78d9-8e06-4b43-b17a-ec37334979bc-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:39:55.962792 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.962778    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbff78d9-8e06-4b43-b17a-ec37334979bc" (UID: "dbff78d9-8e06-4b43-b17a-ec37334979bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:39:55.964397 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.964367    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbff78d9-8e06-4b43-b17a-ec37334979bc" (UID: "dbff78d9-8e06-4b43-b17a-ec37334979bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:39:55.964499 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:55.964443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-api-access-5gt28" (OuterVolumeSpecName: "kube-api-access-5gt28") pod "dbff78d9-8e06-4b43-b17a-ec37334979bc" (UID: "dbff78d9-8e06-4b43-b17a-ec37334979bc"). InnerVolumeSpecName "kube-api-access-5gt28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:39:56.063302 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.063224    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gt28\" (UniqueName: \"kubernetes.io/projected/dbff78d9-8e06-4b43-b17a-ec37334979bc-kube-api-access-5gt28\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:39:56.063302 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.063253    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbff78d9-8e06-4b43-b17a-ec37334979bc-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:39:56.063302 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.063267    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbff78d9-8e06-4b43-b17a-ec37334979bc-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:39:56.103206 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.103177    2580 generic.go:356] "Generic (PLEG): container finished" podID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerID="8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d" exitCode=0
Sep 11 16:39:56.103317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.103218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerDied","Data":"8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d"}
Sep 11 16:39:56.103317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.103233    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"
Sep 11 16:39:56.103317 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.103299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp" event={"ID":"dbff78d9-8e06-4b43-b17a-ec37334979bc","Type":"ContainerDied","Data":"d773c97a9aff258f580e33158a2b8ad66bb38d2b124939038cabe386378dc71e"}
Sep 11 16:39:56.103432 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.103324    2580 scope.go:117] "RemoveContainer" containerID="9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106"
Sep 11 16:39:56.112148 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.112128    2580 scope.go:117] "RemoveContainer" containerID="8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d"
Sep 11 16:39:56.119343 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.119326    2580 scope.go:117] "RemoveContainer" containerID="5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5"
Sep 11 16:39:56.126002 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.125981    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"]
Sep 11 16:39:56.126620 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.126606    2580 scope.go:117] "RemoveContainer" containerID="9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106"
Sep 11 16:39:56.126886 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:39:56.126865    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106\": container with ID starting with 9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106 not found: ID does not exist" containerID="9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106"
Sep 11 16:39:56.126953 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.126894    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106"} err="failed to get container status \"9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106\": rpc error: code = NotFound desc = could not find container \"9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106\": container with ID starting with 9d9cffe2c7e22eb27b6e7fd4b20fa89d335b67373001b73ce9508ac000132106 not found: ID does not exist"
Sep 11 16:39:56.126953 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.126911    2580 scope.go:117] "RemoveContainer" containerID="8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d"
Sep 11 16:39:56.127133 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:39:56.127117    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d\": container with ID starting with 8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d not found: ID does not exist" containerID="8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d"
Sep 11 16:39:56.127174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.127140    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d"} err="failed to get container status \"8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d\": rpc error: code = NotFound desc = could not find container \"8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d\": container with ID starting with 8930734afed5b8a54c20a5aba3f7f409117c74cb9b27f425ac9a9d019f6e327d not found: ID does not exist"
Sep 11 16:39:56.127174 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.127153    2580 scope.go:117] "RemoveContainer" containerID="5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5"
Sep 11 16:39:56.127357 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:39:56.127343    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5\": container with ID starting with 5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5 not found: ID does not exist" containerID="5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5"
Sep 11 16:39:56.127411 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.127360    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5"} err="failed to get container status \"5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5\": rpc error: code = NotFound desc = could not find container \"5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5\": container with ID starting with 5e6856bc4b6cd11cbf04449c4b355c4d3da92d3752469a256e397b3256825af5 not found: ID does not exist"
Sep 11 16:39:56.132246 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.132224    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jdmdp"]
Sep 11 16:39:56.939435 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:39:56.939404    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" path="/var/lib/kubelet/pods/dbff78d9-8e06-4b43-b17a-ec37334979bc/volumes"
Sep 11 16:41:21.850575 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:21.850533    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"]
Sep 11 16:41:21.851140 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:21.850864    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" containerID="cri-o://1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935" gracePeriod=30
Sep 11 16:41:21.851140 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:21.850927    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kube-rbac-proxy" containerID="cri-o://c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02" gracePeriod=30
Sep 11 16:41:21.891658 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:21.891630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:22.460580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:22.457986    2580 generic.go:356] "Generic (PLEG): container finished" podID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerID="c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02" exitCode=2
Sep 11 16:41:22.460580 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:22.458071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerDied","Data":"c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02"}
Sep 11 16:41:23.718112 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:23.718073    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 11 16:41:23.723382 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:23.723357    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:41:26.294464 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.294438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:41:26.343672 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.343583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5be48400-8ded-4271-90a0-9d7901ad77ae-kserve-provision-location\") pod \"5be48400-8ded-4271-90a0-9d7901ad77ae\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") "
Sep 11 16:41:26.343672 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.343639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-cabundle-cert\") pod \"5be48400-8ded-4271-90a0-9d7901ad77ae\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") "
Sep 11 16:41:26.343922 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.343702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls\") pod \"5be48400-8ded-4271-90a0-9d7901ad77ae\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") "
Sep 11 16:41:26.343922 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.343737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-rbac-proxy-sar-config\") pod \"5be48400-8ded-4271-90a0-9d7901ad77ae\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") "
Sep 11 16:41:26.343922 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.343773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm9cx\" (UniqueName: \"kubernetes.io/projected/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-api-access-pm9cx\") pod \"5be48400-8ded-4271-90a0-9d7901ad77ae\" (UID: \"5be48400-8ded-4271-90a0-9d7901ad77ae\") "
Sep 11 16:41:26.344053 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.343934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be48400-8ded-4271-90a0-9d7901ad77ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5be48400-8ded-4271-90a0-9d7901ad77ae" (UID: "5be48400-8ded-4271-90a0-9d7901ad77ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:26.344119 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.344093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5be48400-8ded-4271-90a0-9d7901ad77ae" (UID: "5be48400-8ded-4271-90a0-9d7901ad77ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:41:26.344170 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.344147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5be48400-8ded-4271-90a0-9d7901ad77ae" (UID: "5be48400-8ded-4271-90a0-9d7901ad77ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:41:26.345935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.345901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-api-access-pm9cx" (OuterVolumeSpecName: "kube-api-access-pm9cx") pod "5be48400-8ded-4271-90a0-9d7901ad77ae" (UID: "5be48400-8ded-4271-90a0-9d7901ad77ae"). InnerVolumeSpecName "kube-api-access-pm9cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:41:26.345935 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.345922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5be48400-8ded-4271-90a0-9d7901ad77ae" (UID: "5be48400-8ded-4271-90a0-9d7901ad77ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:41:26.445061 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.445027    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9cx\" (UniqueName: \"kubernetes.io/projected/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-api-access-pm9cx\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:41:26.445061 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.445054    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5be48400-8ded-4271-90a0-9d7901ad77ae-kserve-provision-location\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:41:26.445061 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.445065    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-cabundle-cert\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:41:26.445061 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.445073    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5be48400-8ded-4271-90a0-9d7901ad77ae-proxy-tls\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:41:26.445319 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.445081    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5be48400-8ded-4271-90a0-9d7901ad77ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:41:26.478314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.478284    2580 generic.go:356] "Generic (PLEG): container finished" podID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerID="1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935" exitCode=0
Sep 11 16:41:26.478466 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.478341    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"
Sep 11 16:41:26.478466 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.478369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerDied","Data":"1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935"}
Sep 11 16:41:26.478466 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.478437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h" event={"ID":"5be48400-8ded-4271-90a0-9d7901ad77ae","Type":"ContainerDied","Data":"764f6596d5e7a95f658eefc8e7c65248e817645da65f6af369c7b387f5b8b06e"}
Sep 11 16:41:26.478578 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.478469    2580 scope.go:117] "RemoveContainer" containerID="c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02"
Sep 11 16:41:26.487339 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.487322    2580 scope.go:117] "RemoveContainer" containerID="1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935"
Sep 11 16:41:26.494692 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.494676    2580 scope.go:117] "RemoveContainer" containerID="57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294"
Sep 11 16:41:26.500596 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.500568    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"]
Sep 11 16:41:26.502809 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.502788    2580 scope.go:117] "RemoveContainer" containerID="c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02"
Sep 11 16:41:26.503240 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:41:26.503219    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02\": container with ID starting with c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02 not found: ID does not exist" containerID="c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02"
Sep 11 16:41:26.503333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.503248    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02"} err="failed to get container status \"c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02\": rpc error: code = NotFound desc = could not find container \"c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02\": container with ID starting with c6d09e23bf544eaba110cef7f8941b6c44bf729e2cf5f86dbd33e03b9d4c6b02 not found: ID does not exist"
Sep 11 16:41:26.503333 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.503266    2580 scope.go:117] "RemoveContainer" containerID="1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935"
Sep 11 16:41:26.503539 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:41:26.503522    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935\": container with ID starting with 1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935 not found: ID does not exist" containerID="1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935"
Sep 11 16:41:26.503603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.503542    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935"} err="failed to get container status \"1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935\": rpc error: code = NotFound desc = could not find container \"1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935\": container with ID starting with 1f3f95992a6734aa88d8d10f1eb80406f7cd7719f9d4c02b1b59e93b331a6935 not found: ID does not exist"
Sep 11 16:41:26.503603 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.503553    2580 scope.go:117] "RemoveContainer" containerID="57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294"
Sep 11 16:41:26.503797 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:41:26.503779    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294\": container with ID starting with 57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294 not found: ID does not exist" containerID="57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294"
Sep 11 16:41:26.503878 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.503802    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294"} err="failed to get container status \"57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294\": rpc error: code = NotFound desc = could not find container \"57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294\": container with ID starting with 57ce0b040aa6adbd9ab4437ad1b6246fd06a282c13943ef8324fcc71fd5ba294 not found: ID does not exist"
Sep 11 16:41:26.504699 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.504677    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-22x6h"]
Sep 11 16:41:26.939372 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:26.939339    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" path="/var/lib/kubelet/pods/5be48400-8ded-4271-90a0-9d7901ad77ae/volumes"
Sep 11 16:41:35.802302 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:35.802267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:35.983541 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:35.983510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:46.724837 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:46.724791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:46.778871 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:41:46.778818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:39.460080 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:42:39.460030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:18.980890 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:18.980850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:21.204496 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:21.204461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:21.269567 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:21.269530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:22.696521 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696490    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8pb5/must-gather-rdt6n"]
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696806    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="storage-initializer"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696818    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="storage-initializer"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696856    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="storage-initializer"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696866    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="storage-initializer"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696876    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kube-rbac-proxy"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696883    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kube-rbac-proxy"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696896    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kube-rbac-proxy"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696901    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kube-rbac-proxy"
Sep 11 16:43:22.696908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696912    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696918    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696924    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696929    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696981    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kserve-container"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696989    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kube-rbac-proxy"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.696997    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be48400-8ded-4271-90a0-9d7901ad77ae" containerName="kserve-container"
Sep 11 16:43:22.697201 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.697003    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbff78d9-8e06-4b43-b17a-ec37334979bc" containerName="kube-rbac-proxy"
Sep 11 16:43:22.700107 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.700075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:22.702616 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.702590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k8pb5\"/\"default-dockercfg-spwzk\""
Sep 11 16:43:22.702744 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.702728    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8pb5\"/\"openshift-service-ca.crt\""
Sep 11 16:43:22.703020 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.702979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8pb5\"/\"kube-root-ca.crt\""
Sep 11 16:43:22.707530 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.707508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:22.708132 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.708110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8pb5/must-gather-rdt6n"]
Sep 11 16:43:22.766407 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.766365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:22.866446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.866399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eec34f44-1abd-43bb-acd2-d3e3ad50e489-must-gather-output\") pod \"must-gather-rdt6n\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") " pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:22.866446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.866449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wg8\" (UniqueName: \"kubernetes.io/projected/eec34f44-1abd-43bb-acd2-d3e3ad50e489-kube-api-access-29wg8\") pod \"must-gather-rdt6n\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") " pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:22.967134 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.967040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eec34f44-1abd-43bb-acd2-d3e3ad50e489-must-gather-output\") pod \"must-gather-rdt6n\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") " pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:22.967134 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.967083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wg8\" (UniqueName: \"kubernetes.io/projected/eec34f44-1abd-43bb-acd2-d3e3ad50e489-kube-api-access-29wg8\") pod \"must-gather-rdt6n\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") " pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:22.967438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.967414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eec34f44-1abd-43bb-acd2-d3e3ad50e489-must-gather-output\") pod \"must-gather-rdt6n\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") " pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:22.975952 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:22.975917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wg8\" (UniqueName: \"kubernetes.io/projected/eec34f44-1abd-43bb-acd2-d3e3ad50e489-kube-api-access-29wg8\") pod \"must-gather-rdt6n\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") " pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:23.017984 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:23.017950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:23.286438 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:23.286400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8pb5/must-gather-rdt6n"]
Sep 11 16:43:23.288527 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:43:23.288498    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec34f44_1abd_43bb_acd2_d3e3ad50e489.slice/crio-598c8279555e3fd032788e148942006112586f27b78dfdbb6fc92c25eb132069 WatchSource:0}: Error finding container 598c8279555e3fd032788e148942006112586f27b78dfdbb6fc92c25eb132069: Status 404 returned error can't find the container with id 598c8279555e3fd032788e148942006112586f27b78dfdbb6fc92c25eb132069
Sep 11 16:43:23.290265 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:23.290249    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:43:23.955379 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:23.955334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8pb5/must-gather-rdt6n" event={"ID":"eec34f44-1abd-43bb-acd2-d3e3ad50e489","Type":"ContainerStarted","Data":"598c8279555e3fd032788e148942006112586f27b78dfdbb6fc92c25eb132069"}
Sep 11 16:43:29.986984 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:29.986897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8pb5/must-gather-rdt6n" event={"ID":"eec34f44-1abd-43bb-acd2-d3e3ad50e489","Type":"ContainerStarted","Data":"11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967"}
Sep 11 16:43:29.986984 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:29.986939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8pb5/must-gather-rdt6n" event={"ID":"eec34f44-1abd-43bb-acd2-d3e3ad50e489","Type":"ContainerStarted","Data":"a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"}
Sep 11 16:43:30.027370 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:30.027309    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k8pb5/must-gather-rdt6n" podStartSLOduration=1.890975474 podStartE2EDuration="8.027291869s" totalImagesPullingTime="6.136316395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:30.026291482 +0000 UTC m=+1585.707395908" watchObservedRunningTime="2026-09-11 16:43:30.027291869 +0000 UTC m=+1585.708396294"
Sep 11 16:43:30.456563 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:30.456528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:53.095281 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:53.095249    2580 generic.go:356] "Generic (PLEG): container finished" podID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerID="a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4" exitCode=0
Sep 11 16:43:53.095665 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:53.095318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8pb5/must-gather-rdt6n" event={"ID":"eec34f44-1abd-43bb-acd2-d3e3ad50e489","Type":"ContainerDied","Data":"a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"}
Sep 11 16:43:53.095665 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:53.095634    2580 scope.go:117] "RemoveContainer" containerID="a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"
Sep 11 16:43:53.330153 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:53.330117    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k8pb5_must-gather-rdt6n_eec34f44-1abd-43bb-acd2-d3e3ad50e489/gather/0.log"
Sep 11 16:43:53.794689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:53.794592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:53.943367 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:53.943328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:54.042426 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:54.042394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:56.808957 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:56.808909    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gmprr_1c159bf6-5095-4cf6-b9fc-0935e2fd00ca/global-pull-secret-syncer/0.log"
Sep 11 16:43:56.957577 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:56.957513    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log"
Sep 11 16:43:57.101039 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:57.100955    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-32.ec2.internal_20e5ec496ea128525a2252a4ae0e8d2f/haproxy/0.log"
Sep 11 16:43:58.768444 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:58.768408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-k8pb5/must-gather-rdt6n"]
Sep 11 16:43:58.768977 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:58.768618    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-k8pb5/must-gather-rdt6n" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="copy" containerID="cri-o://11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967" gracePeriod=2
Sep 11 16:43:58.773353 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:58.773327    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-k8pb5/must-gather-rdt6n"]
Sep 11 16:43:59.003524 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.003498    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k8pb5_must-gather-rdt6n_eec34f44-1abd-43bb-acd2-d3e3ad50e489/copy/0.log"
Sep 11 16:43:59.005104 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.005086    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:59.086908 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.086807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29wg8\" (UniqueName: \"kubernetes.io/projected/eec34f44-1abd-43bb-acd2-d3e3ad50e489-kube-api-access-29wg8\") pod \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") "
Sep 11 16:43:59.087043 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.086935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eec34f44-1abd-43bb-acd2-d3e3ad50e489-must-gather-output\") pod \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\" (UID: \"eec34f44-1abd-43bb-acd2-d3e3ad50e489\") "
Sep 11 16:43:59.088927 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.088904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec34f44-1abd-43bb-acd2-d3e3ad50e489-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "eec34f44-1abd-43bb-acd2-d3e3ad50e489" (UID: "eec34f44-1abd-43bb-acd2-d3e3ad50e489"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:59.089056 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.089041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec34f44-1abd-43bb-acd2-d3e3ad50e489-kube-api-access-29wg8" (OuterVolumeSpecName: "kube-api-access-29wg8") pod "eec34f44-1abd-43bb-acd2-d3e3ad50e489" (UID: "eec34f44-1abd-43bb-acd2-d3e3ad50e489"). InnerVolumeSpecName "kube-api-access-29wg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:59.118656 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.118629    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k8pb5_must-gather-rdt6n_eec34f44-1abd-43bb-acd2-d3e3ad50e489/copy/0.log"
Sep 11 16:43:59.119017 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.118998    2580 generic.go:356] "Generic (PLEG): container finished" podID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerID="11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967" exitCode=143
Sep 11 16:43:59.119098 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.119053    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8pb5/must-gather-rdt6n"
Sep 11 16:43:59.119136 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.119098    2580 scope.go:117] "RemoveContainer" containerID="11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967"
Sep 11 16:43:59.126982 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.126960    2580 scope.go:117] "RemoveContainer" containerID="a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"
Sep 11 16:43:59.140573 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.140555    2580 scope.go:117] "RemoveContainer" containerID="11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967"
Sep 11 16:43:59.140914 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:43:59.140891    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967\": container with ID starting with 11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967 not found: ID does not exist" containerID="11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967"
Sep 11 16:43:59.140973 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.140928    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967"} err="failed to get container status \"11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967\": rpc error: code = NotFound desc = could not find container \"11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967\": container with ID starting with 11586d7501f3f9c9295c3ce7747759da6c7b9f924abb326ad65ea8c0f6a48967 not found: ID does not exist"
Sep 11 16:43:59.140973 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.140948    2580 scope.go:117] "RemoveContainer" containerID="a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"
Sep 11 16:43:59.141195 ip-10-0-137-32 kubenswrapper[2580]: E0911 16:43:59.141176    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4\": container with ID starting with a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4 not found: ID does not exist" containerID="a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"
Sep 11 16:43:59.141251 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.141203    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4"} err="failed to get container status \"a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4\": rpc error: code = NotFound desc = could not find container \"a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4\": container with ID starting with a7a388f8c86ffda1790251123c082f9552f5cbb175f8d2aff58b2bfe0e60b6f4 not found: ID does not exist"
Sep 11 16:43:59.188478 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.188390    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wg8\" (UniqueName: \"kubernetes.io/projected/eec34f44-1abd-43bb-acd2-d3e3ad50e489-kube-api-access-29wg8\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:43:59.188478 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.188418    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eec34f44-1abd-43bb-acd2-d3e3ad50e489-must-gather-output\") on node \"ip-10-0-137-32.ec2.internal\" DevicePath \"\""
Sep 11 16:43:59.636570 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.636533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:59.697083 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:43:59.697041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wl27_b264b7cd-f54d-4f10-b644-22fc59c4ab43/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:00.940800 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:00.940762    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" path="/var/lib/kubelet/pods/eec34f44-1abd-43bb-acd2-d3e3ad50e489/volumes"
Sep 11 16:44:01.404619 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.404585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wxxcr_67afa78c-a699-4ec5-ae58-68f01f025fa7/kube-state-metrics/0.log"
Sep 11 16:44:01.429314 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.429280    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wxxcr_67afa78c-a699-4ec5-ae58-68f01f025fa7/kube-rbac-proxy-main/0.log"
Sep 11 16:44:01.455425 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.455379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wxxcr_67afa78c-a699-4ec5-ae58-68f01f025fa7/kube-rbac-proxy-self/0.log"
Sep 11 16:44:01.486082 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.486041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d564766cb-khndb_aa850e6a-27cc-482e-959b-81ed5b525dc1/metrics-server/0.log"
Sep 11 16:44:01.510693 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.510671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-ln6lc_38c9fdc6-9d85-47fe-9f8e-4ec0527bf15b/monitoring-plugin/0.log"
Sep 11 16:44:01.703422 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.703348    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzzrn_4bea74c9-0f80-47a9-9625-17ca6c25cb4a/node-exporter/0.log"
Sep 11 16:44:01.731047 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.731019    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzzrn_4bea74c9-0f80-47a9-9625-17ca6c25cb4a/kube-rbac-proxy/0.log"
Sep 11 16:44:01.753125 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.753082    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzzrn_4bea74c9-0f80-47a9-9625-17ca6c25cb4a/init-textfile/0.log"
Sep 11 16:44:01.779127 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.779101    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vfwmr_a61163c6-6f55-4aab-be6e-14b682c63639/kube-rbac-proxy-main/0.log"
Sep 11 16:44:01.799477 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.799458    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vfwmr_a61163c6-6f55-4aab-be6e-14b682c63639/kube-rbac-proxy-self/0.log"
Sep 11 16:44:01.820316 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.820295    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vfwmr_a61163c6-6f55-4aab-be6e-14b682c63639/openshift-state-metrics/0.log"
Sep 11 16:44:01.872736 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.872710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/prometheus/0.log"
Sep 11 16:44:01.889304 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.889282    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/config-reloader/0.log"
Sep 11 16:44:01.915934 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.915911    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/thanos-sidecar/0.log"
Sep 11 16:44:01.936419 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.936400    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/kube-rbac-proxy-web/0.log"
Sep 11 16:44:01.958016 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.957965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/kube-rbac-proxy/0.log"
Sep 11 16:44:01.982465 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:01.982439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/kube-rbac-proxy-thanos/0.log"
Sep 11 16:44:02.003761 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.003740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4755f6f6-794c-48a1-aee0-0f9b45e9718d/init-config-reloader/0.log"
Sep 11 16:44:02.032718 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.032689    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-24hv8_59c4f32b-34e6-479e-9005-d20271b23e28/prometheus-operator/0.log"
Sep 11 16:44:02.050682 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.050662    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-24hv8_59c4f32b-34e6-479e-9005-d20271b23e28/kube-rbac-proxy/0.log"
Sep 11 16:44:02.075123 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.075100    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-d5gxx_bb0d6381-a374-4372-b711-9ebe6700d403/prometheus-operator-admission-webhook/0.log"
Sep 11 16:44:02.105115 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.105092    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56bd74bfd8-kjg7s_569e6a2c-6c78-4982-b64c-9dc201fe0510/telemeter-client/0.log"
Sep 11 16:44:02.125571 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.125551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56bd74bfd8-kjg7s_569e6a2c-6c78-4982-b64c-9dc201fe0510/reload/0.log"
Sep 11 16:44:02.148280 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.148260    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56bd74bfd8-kjg7s_569e6a2c-6c78-4982-b64c-9dc201fe0510/kube-rbac-proxy/0.log"
Sep 11 16:44:02.186275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.186243    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75956f6695-ckhj7_de14faed-a419-4628-a3b6-3695750ef60f/thanos-query/0.log"
Sep 11 16:44:02.210490 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.210430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75956f6695-ckhj7_de14faed-a419-4628-a3b6-3695750ef60f/kube-rbac-proxy-web/0.log"
Sep 11 16:44:02.230669 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.230646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75956f6695-ckhj7_de14faed-a419-4628-a3b6-3695750ef60f/kube-rbac-proxy/0.log"
Sep 11 16:44:02.252923 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.252903    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75956f6695-ckhj7_de14faed-a419-4628-a3b6-3695750ef60f/prom-label-proxy/0.log"
Sep 11 16:44:02.281100 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.281080    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75956f6695-ckhj7_de14faed-a419-4628-a3b6-3695750ef60f/kube-rbac-proxy-rules/0.log"
Sep 11 16:44:02.301949 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:02.301927    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75956f6695-ckhj7_de14faed-a419-4628-a3b6-3695750ef60f/kube-rbac-proxy-metrics/0.log"
Sep 11 16:44:04.639284 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:04.639249    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6486675c9b-tdvjw_e01ed241-b08e-4be5-8a25-c3b0b7cfa706/console/0.log"
Sep 11 16:44:04.674910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:04.674882    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-r7ffb_26439149-ee7a-46ec-8eaf-8544b10dab09/download-server/0.log"
Sep 11 16:44:05.396926 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.396887    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"]
Sep 11 16:44:05.397231 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.397219    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="copy"
Sep 11 16:44:05.397275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.397232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="copy"
Sep 11 16:44:05.397275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.397247    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="gather"
Sep 11 16:44:05.397275 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.397252    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="gather"
Sep 11 16:44:05.397377 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.397301    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="gather"
Sep 11 16:44:05.397377 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.397310    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec34f44-1abd-43bb-acd2-d3e3ad50e489" containerName="copy"
Sep 11 16:44:05.401381 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.401361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.404554 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.404535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9j26t\"/\"default-dockercfg-9ckph\""
Sep 11 16:44:05.404689 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.404559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9j26t\"/\"openshift-service-ca.crt\""
Sep 11 16:44:05.405749 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.405728    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9j26t\"/\"kube-root-ca.crt\""
Sep 11 16:44:05.407548 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.407528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"]
Sep 11 16:44:05.545290 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.545250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-podres\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.545290 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.545290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-proc\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.545547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.545313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6sr\" (UniqueName: \"kubernetes.io/projected/2a937975-4a3f-4cba-95dc-47002ec5854c-kube-api-access-4v6sr\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.545547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.545411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-sys\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.545547 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.545462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-lib-modules\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.646842 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.646798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-lib-modules\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.646896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-podres\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.646913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-proc\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.646932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6sr\" (UniqueName: \"kubernetes.io/projected/2a937975-4a3f-4cba-95dc-47002ec5854c-kube-api-access-4v6sr\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.647008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-proc\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.647020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-lib-modules\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.647046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-podres\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.647054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-sys\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.647342 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.647136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a937975-4a3f-4cba-95dc-47002ec5854c-sys\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.655484 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.655449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6sr\" (UniqueName: \"kubernetes.io/projected/2a937975-4a3f-4cba-95dc-47002ec5854c-kube-api-access-4v6sr\") pod \"perf-node-gather-daemonset-56ggl\" (UID: \"2a937975-4a3f-4cba-95dc-47002ec5854c\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.712573 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.712519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:05.980301 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:05.980273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"]
Sep 11 16:44:05.982741 ip-10-0-137-32 kubenswrapper[2580]: W0911 16:44:05.982709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a937975_4a3f_4cba_95dc_47002ec5854c.slice/crio-843f7e5fa957d091dc6b3ba46a69f9e55978cd495186bb9ece39ab3c2d8af031 WatchSource:0}: Error finding container 843f7e5fa957d091dc6b3ba46a69f9e55978cd495186bb9ece39ab3c2d8af031: Status 404 returned error can't find the container with id 843f7e5fa957d091dc6b3ba46a69f9e55978cd495186bb9ece39ab3c2d8af031
Sep 11 16:44:06.150835 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.150785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl" event={"ID":"2a937975-4a3f-4cba-95dc-47002ec5854c","Type":"ContainerStarted","Data":"7b57f19296ab7ad155a0dd7098c3c9c425d2bdaaed0392edf1f67bde79addf17"}
Sep 11 16:44:06.151023 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.150854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl" event={"ID":"2a937975-4a3f-4cba-95dc-47002ec5854c","Type":"ContainerStarted","Data":"843f7e5fa957d091dc6b3ba46a69f9e55978cd495186bb9ece39ab3c2d8af031"}
Sep 11 16:44:06.151023 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.150918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:06.166924 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.166876    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl" podStartSLOduration=1.166859817 podStartE2EDuration="1.166859817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:06.166341825 +0000 UTC m=+1621.847446251" watchObservedRunningTime="2026-09-11 16:44:06.166859817 +0000 UTC m=+1621.847964239"
Sep 11 16:44:06.199972 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.199947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzs9s_b4bfc682-6484-4657-9d87-e47fa86e088a/dns/0.log"
Sep 11 16:44:06.222926 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.222903    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzs9s_b4bfc682-6484-4657-9d87-e47fa86e088a/kube-rbac-proxy/0.log"
Sep 11 16:44:06.296506 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.296434    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5fvg_1e9c8529-29e0-4e00-b061-becbd28e2f1b/dns-node-resolver/0.log"
Sep 11 16:44:06.805938 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.805900    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76485bf8fb-wn22k_a0abb8f7-a72a-45df-8a45-9c0adc0b220e/registry/0.log"
Sep 11 16:44:06.853997 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:06.853960    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qm7s8_7d65744c-640a-47cf-bd5d-966db47417af/node-ca/0.log"
Sep 11 16:44:07.991898 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:07.991847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnwsz_82e9b905-460b-4c58-b2ae-c64e8c4fa5d3/serve-healthcheck-canary/0.log"
Sep 11 16:44:08.526910 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:08.526876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7dmz_56f5fd17-90aa-42e8-9989-6cfa74f86449/kube-rbac-proxy/0.log"
Sep 11 16:44:08.547813 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:08.547783    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7dmz_56f5fd17-90aa-42e8-9989-6cfa74f86449/exporter/0.log"
Sep 11 16:44:08.568630 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:08.568606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7dmz_56f5fd17-90aa-42e8-9989-6cfa74f86449/extractor/0.log"
Sep 11 16:44:11.239446 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:11.239411    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7nxrf_219f8569-40dc-4d3d-b751-5c3dee1d2e20/s3-init/0.log"
Sep 11 16:44:11.257562 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:11.257528    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7nxrf_219f8569-40dc-4d3d-b751-5c3dee1d2e20/download-hf-model/0.log"
Sep 11 16:44:12.164044 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:12.164019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-56ggl"
Sep 11 16:44:16.772569 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:16.772537    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5b6cb_c764ae30-1eb4-43a7-a78e-bd975b3fa3e3/kube-multus/0.log"
Sep 11 16:44:16.939946 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:16.939922    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/kube-multus-additional-cni-plugins/0.log"
Sep 11 16:44:16.960635 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:16.960611    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/egress-router-binary-copy/0.log"
Sep 11 16:44:16.981465 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:16.981441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/cni-plugins/0.log"
Sep 11 16:44:17.001341 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:17.001320    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/bond-cni-plugin/0.log"
Sep 11 16:44:17.025556 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:17.025487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/routeoverride-cni/0.log"
Sep 11 16:44:17.047033 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:17.047004    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/whereabouts-cni-bincopy/0.log"
Sep 11 16:44:17.070291 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:17.070266    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jhltr_58c23413-8a02-469b-bf68-39ddd5729968/whereabouts-cni/0.log"
Sep 11 16:44:17.337452 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:17.337370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lhpp4_8cd81b90-21f6-4343-99c5-3d746e3c480e/network-metrics-daemon/0.log"
Sep 11 16:44:17.356439 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:17.356410    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lhpp4_8cd81b90-21f6-4343-99c5-3d746e3c480e/kube-rbac-proxy/0.log"
Sep 11 16:44:18.619936 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.619903    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/ovn-controller/0.log"
Sep 11 16:44:18.635377 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.635350    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/ovn-acl-logging/0.log"
Sep 11 16:44:18.659491 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.659461    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/kube-rbac-proxy-node/0.log"
Sep 11 16:44:18.681883 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.681852    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 16:44:18.700237 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.700209    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/northd/0.log"
Sep 11 16:44:18.719875 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.719846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/nbdb/0.log"
Sep 11 16:44:18.744090 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.744061    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/sbdb/0.log"
Sep 11 16:44:18.799805 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:18.799777    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s9f5g_d91b4ccf-9425-4777-9bac-32be72517058/ovnkube-controller/0.log"
Sep 11 16:44:19.868102 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:19.868072    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bdf6j_c0eb169c-3c31-48f2-a973-53f0aacced29/network-check-target-container/0.log"
Sep 11 16:44:20.959044 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:20.959012    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-cnqfb_3da9490e-eee0-48dc-b34e-d00efe02064f/iptables-alerter/0.log"
Sep 11 16:44:21.719006 ip-10-0-137-32 kubenswrapper[2580]: I0911 16:44:21.718971    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-cnvx5_a073bb01-4ab4-4f26-aa5d-3f6eca7c2ae4/tuned/0.log"