Sep 19 08:20:47.814979 ip-10-0-140-141 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 08:20:51.355240 ip-10-0-140-141 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:51.355240 ip-10-0-140-141 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 08:20:51.355240 ip-10-0-140-141 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:51.355240 ip-10-0-140-141 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 08:20:51.355240 ip-10-0-140-141 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:51.356038 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.355307    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 08:20:51.358275 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358258    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:51.358275 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358274    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358280    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358284    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358288    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358291    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358294    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358297    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358300    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358304    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358307    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358309    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358312    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358315    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358318    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358321    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358323    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358326    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358329    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358333    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:51.358338 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358336    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358339    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358342    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358350    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358353    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358356    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358359    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358362    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358365    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358367    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358370    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358373    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358376    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358379    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358381    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358384    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358387    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358390    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358392    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358395    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:51.358818 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358397    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358400    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358402    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358405    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358407    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358410    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358412    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358415    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358417    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358420    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358423    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358425    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358428    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358430    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358434    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358437    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358440    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358442    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358445    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358447    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:51.359317 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358450    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358452    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358455    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358457    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358460    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358462    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358465    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358468    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358470    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358473    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358476    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358478    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358481    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358483    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358486    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358490    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358492    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358495    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358498    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358500    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:51.359808 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358503    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358505    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358508    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358510    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358513    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358519    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358523    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358526    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358529    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358531    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358534    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358536    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358540    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358544    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358547    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358550    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358552    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358555    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.358557    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359277    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:51.360283 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359284    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359288    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359291    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359294    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359297    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359300    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359303    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359305    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359310    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359313    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359315    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359318    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359321    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359324    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359327    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359329    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359333    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359335    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359339    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359342    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:51.360789 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359344    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359347    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359350    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359352    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359355    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359358    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359360    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359363    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359366    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359368    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359371    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359373    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359376    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359378    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359381    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359384    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359386    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359389    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359391    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359394    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:51.361285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359397    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359400    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359403    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359406    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359408    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359412    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359415    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359417    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359420    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359423    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359425    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359428    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359430    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359433    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359436    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359439    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359441    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359445    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359449    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359452    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:51.361799 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359455    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359458    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359460    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359463    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359465    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359468    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359471    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359474    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359478    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359481    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359483    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359487    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359491    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359494    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359499    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359503    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359506    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359508    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359511    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:51.362285 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359513    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359516    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359519    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359521    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359524    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359527    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359529    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359532    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359535    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359537    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359540    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359542    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359545    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359548    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359550    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359553    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359555    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359558    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359560    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:51.362765 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359939    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359945    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359948    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359951    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359954    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359956    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359959    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359962    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359967    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359970    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359973    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359976    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359978    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359983    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359987    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359990    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359993    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359996    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.359999    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:51.363249 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360002    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360005    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360007    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360010    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360013    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360015    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360018    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360021    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360023    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360026    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360028    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360031    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360033    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360036    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360038    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360041    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360043    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360046    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360049    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360052    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:51.363708 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360054    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360058    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360061    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360064    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360066    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360069    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360072    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360074    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360077    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360079    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360082    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360084    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360087    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360090    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360093    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360095    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360098    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360100    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360103    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360105    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:51.364200 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360108    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360111    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360113    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360116    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360118    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360121    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360123    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360126    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360129    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360131    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360134    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360137    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360140    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360142    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360145    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360149    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360152    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360156    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360160    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360163    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:51.364745 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360165    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360168    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360172    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360175    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360178    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360181    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360184    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360186    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360189    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360191    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360194    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360197    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360199    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360202    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360204    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360207    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360210    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360215    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360218    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.360220    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:51.365220 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360301    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360342    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360911    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360917    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360921    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360925    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360929    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360933    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360940    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360944    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360947    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360950    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360953    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360957    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360960    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360963    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360966    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360969    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360974    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360977    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360980    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360983    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360987    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360991    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 08:20:51.365708 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360994    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.360997    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361001    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361004    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361007    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361013    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361016    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361019    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361023    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361026    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361029    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361032    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361035    2579 flags.go:64] FLAG: --enable-server="true"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361038    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361042    2579 flags.go:64] FLAG: --event-burst="100"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361046    2579 flags.go:64] FLAG: --event-qps="50"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361049    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361052    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361055    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361059    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361062    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361065    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361068    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361071    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361073    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 08:20:51.366293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361077    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361080    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361083    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361086    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361089    2579 flags.go:64] FLAG: --feature-gates=""
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361093    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361096    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361100    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361102    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361106    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361109    2579 flags.go:64] FLAG: --help="false"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361112    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361116    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361119    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361122    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361126    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361129    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361132    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361135    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361138    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361141    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361144    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361147    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361150    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 19 08:20:51.366908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361153    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361156    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361160    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361163    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361166    2579 flags.go:64] FLAG: --lock-file=""
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361168    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361171    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361175    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361180    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361183    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361186    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361189    2579 flags.go:64] FLAG: --logging-format="text"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361192    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361196    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361198    2579 flags.go:64] FLAG: --manifest-url=""
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361201    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361206    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361209    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361213    2579 flags.go:64] FLAG: --max-pods="110"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361216    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361220    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361223    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361226    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361229    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361232    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 08:20:51.367481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361235    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361243    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361246    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361249    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361252    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361255    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361261    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361264    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361267    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361271    2579 flags.go:64] FLAG: --port="10250"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361274    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361277    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dcd96f358632d2be"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361280    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361283    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361286    2579 flags.go:64] FLAG: --register-node="true"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361289    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361293    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361296    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361299    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361302    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361306    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361309    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361312    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361315    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361318    2579 flags.go:64] FLAG: --runonce="false"
Sep 19 08:20:51.368274 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361321    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361324    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361328    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361331    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361334    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361338    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361341    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361344    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361347    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361350    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361353    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361356    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361359    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361362    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361365    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361370    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361373    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361377    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361382    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361384    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361387    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361390    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361393    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361396    2579 flags.go:64] FLAG: --v="2"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361400    2579 flags.go:64] FLAG: --version="false"
Sep 19 08:20:51.369420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361405    2579 flags.go:64] FLAG: --vmodule=""
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361409    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.361412    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361525    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361530    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361533    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361536    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361539    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361542    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361546    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361549    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361552    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361555    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361557    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361561    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361564    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361566    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361569    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361572    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361575    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:51.370487 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361577    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361580    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361583    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361585    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361588    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361591    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361594    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361597    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361600    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361602    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361605    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361608    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361612    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361615    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361617    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361620    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361623    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361626    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361628    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:51.371392 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361632    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361636    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361640    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361644    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361647    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361650    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361653    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361656    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361660    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361664    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361666    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361669    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361672    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361675    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361678    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361680    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361684    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361687    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361701    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361711    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:51.372008 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361714    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361717    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361719    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361722    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361725    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361731    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361734    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361737    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361740    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361743    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361746    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361749    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361751    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361754    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361758    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361761    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361764    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361766    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361769    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:51.372905 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361771    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361775    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361779    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361782    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361785    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361788    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361791    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361793    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361796    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361799    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361801    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361804    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361807    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361816    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361819    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361821    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361824    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361827    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361831    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:51.373668 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361833    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361836    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361839    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361842    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.361844    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.362429    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.370179    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.370198    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.370225    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.370238    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370326    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370333    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370339    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370344    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370348    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:51.374207 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370353    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370357    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370362    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370366    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370371    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370375    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370379    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370384    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370388    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370392    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370396    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370402    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370406    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370411    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370417    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370421    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370426    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370431    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370435    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370440    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:51.374686 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370445    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370449    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370453    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370458    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370463    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370467    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370472    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370476    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370480    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370484    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370489    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370493    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370497    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370501    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370505    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370509    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370513    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370518    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370523    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370528    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370532    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:51.375204 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370537    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370541    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370546    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370551    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370556    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370560    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370564    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370568    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370573    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370577    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370581    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370586    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370590    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370595    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370601    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370609    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370615    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370620    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:51.375790 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370625    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370630    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370634    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370638    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370643    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370647    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370652    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370656    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370660    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370665    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370669    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370674    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370678    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370682    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370686    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370709    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370715    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370719    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370724    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370729    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:51.376327 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370734    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370738    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370742    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370746    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370751    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370755    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370759    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370764    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370768    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370773    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370777    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370781    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370786    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370792    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:51.376833 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370798    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.370806    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370983    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370990    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370995    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.370999    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371004    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371008    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371012    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371017    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371021    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371025    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371029    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371033    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371037    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:51.377214 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371041    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371045    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371050    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371055    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371059    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371063    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371068    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371072    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371077    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371081    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371085    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371089    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371093    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371097    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371102    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371109    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371114    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371119    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371124    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:51.377609 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371129    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371133    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371137    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371141    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371145    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371149    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371153    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371157    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371161    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371166    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371170    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371174    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371178    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371182    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371186    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371191    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371196    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371200    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371204    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371209    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:51.378480 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371213    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371217    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371221    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371225    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371229    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371233    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371238    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371242    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371246    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371251    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371255    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371259    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371263    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371267    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371272    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371276    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371280    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371284    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371289    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371293    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:51.379326 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371297    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371301    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371305    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371309    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371313    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371317    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371323    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371329    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371333    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371338    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371342    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371346    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371350    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371355    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371359    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371366    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371371    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371376    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371381    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:51.380160 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371385    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371390    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371394    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371398    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371402    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371406    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371410    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:51.371414    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.371422    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.371573    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.374247    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.375349    2579 server.go:1019] "Starting client certificate rotation"
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.375453    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:51.380755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.376123    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:51.381125 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.378210    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:51.381125 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.380008    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:51.386038 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.386017    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 19 08:20:51.387522 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.387507    2579 log.go:25] "Validated CRI v1 image API"
Sep 19 08:20:51.393515 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.393499    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 08:20:51.399984 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.399964    2579 fs.go:135] Filesystem UUIDs: map[49f3c832-e615-4722-907b-15871a4dba5d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d70e6438-1c01-4df2-9696-56d19504e5dc:/dev/nvme0n1p3]
Sep 19 08:20:51.400056 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.399982    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 08:20:51.405825 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.405720    2579 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:51.404174119 +0000 UTC m=+2.069309656 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099254 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b4bc39e0b79bdfa11de10eecfb2d6 SystemUUID:ec2b4bc3-9e0b-79bd-fa11-de10eecfb2d6 BootID:ed85e10f-f9d4-491c-9bf9-d6151e08c308 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:df:40:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:df:40:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:9f:44:2d:37:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 08:20:51.405825 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.405822    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 08:20:51.405933 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.405892    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 08:20:51.407444 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.407420    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 08:20:51.407585 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.407444    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-141.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 08:20:51.408522 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.408512    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 08:20:51.408563 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.408524    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 08:20:51.408563 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.408532    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:51.408563 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.408549    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:51.408641 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.408615    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 08:20:51.409003 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.408988    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:51.412080 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412066    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:51.412196 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412186    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 08:20:51.412236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412214    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 08:20:51.412236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412223    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 08:20:51.412291 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412238    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 08:20:51.412291 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412247    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 19 08:20:51.412291 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412257    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 08:20:51.412584 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412569    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:51.412634 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.412591    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:51.416986 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.416966    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 08:20:51.418993 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.418979    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 08:20:51.419037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.419010    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 08:20:51.420596 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420581    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420603    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420610    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420616    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420625    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420630    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420644    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420649    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420656    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 08:20:51.420662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420661    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 08:20:51.420972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420670    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 08:20:51.420972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420675    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 08:20:51.420972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420716    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 08:20:51.420972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.420721    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 08:20:51.421540 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.421507    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-141.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 08:20:51.421604 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.421511    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 08:20:51.424969 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.424957    2579 server.go:1268] "Started kubelet"
Sep 19 08:20:51.425335 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.425316    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 08:20:51.425366 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.425335    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 08:20:51.425417 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.425405    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 08:20:51.425708 ip-10-0-140-141 systemd[1]: Started Kubernetes Kubelet.
Sep 19 08:20:51.426602 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.426585    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 08:20:51.426689 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.426673    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 08:20:51.460990 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.460794    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8r9w5"
Sep 19 08:20:51.461131 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.461106    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8r9w5"
Sep 19 08:20:51.461131 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.461091    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 08:20:51.465159 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.465137    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-141.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 08:20:51.466480 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.465189    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-141.ec2.internal.18d6ab1ef806d243  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-141.ec2.internal,UID:ip-10-0-140-141.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-141.ec2.internal,},FirstTimestamp:2026-09-19 08:20:51.424916035 +0000 UTC m=+2.090051572,LastTimestamp:2026-09-19 08:20:51.424916035 +0000 UTC m=+2.090051572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-141.ec2.internal,}"
Sep 19 08:20:51.467716 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.467704    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 08:20:51.467773 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.467715    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:51.467835 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.467777    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:51.468236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.468217    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 08:20:51.468236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.468237    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 08:20:51.468581 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.468209    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 08:20:51.468581 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.468385    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 08:20:51.468706 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.468592    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 08:20:51.470128 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470111    2579 factory.go:55] Registering systemd factory
Sep 19 08:20:51.470242 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470232    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 19 08:20:51.470729 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470712    2579 factory.go:153] Registering CRI-O factory
Sep 19 08:20:51.470804 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470732    2579 factory.go:223] Registration of the crio container factory successfully
Sep 19 08:20:51.470857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470816    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 08:20:51.470857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470834    2579 factory.go:103] Registering Raw factory
Sep 19 08:20:51.470857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.470847    2579 manager.go:1196] Started watching for new ooms in manager
Sep 19 08:20:51.471279 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.471267    2579 manager.go:349] Starting recovery of all containers
Sep 19 08:20:51.475461 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.475436    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.478707 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.478674    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-141.ec2.internal\" not found" node="ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.479962 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.479947    2579 manager.go:354] Recovery completed
Sep 19 08:20:51.483889 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.483877    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.485612 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.485598    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.485710 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.485630    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.485710 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.485645    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.486160 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486146    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 08:20:51.486160 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486157    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.486248 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486156    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 08:20:51.486296 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486253    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:51.486856 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486842    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.486930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486871    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.486930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.486885    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.487363 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.487350    2579 policy_none.go:49] "None policy: Start"
Sep 19 08:20:51.487401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.487366    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 08:20:51.487401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.487376    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 08:20:51.488382 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.488370    2579 policy_none.go:47] "Start"
Sep 19 08:20:51.534292 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534278    2579 manager.go:337] "Starting Device Plugin manager"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.534320    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534330    2579 server.go:87] "Starting device plugin registration server"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534547    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534567    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534649    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534757    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 08:20:51.535304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.534769    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 08:20:51.535537 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.535309    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 08:20:51.535537 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.535363    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:51.596421 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.596389    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 08:20:51.597504 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.597485    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 08:20:51.597612 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.597532    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 08:20:51.597612 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.597554    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 08:20:51.597612 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.597591    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 08:20:51.599827 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.599806    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.634731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.634636    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.636209 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.636192    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.636285 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.636226    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.636285 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.636239    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.636285 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.636261    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.644204 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.644191    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.644248 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.644212    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-141.ec2.internal\": node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:51.658924 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.658904    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:51.697994 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.697971    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"]
Sep 19 08:20:51.698050 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.698036    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.698973 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.698959    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.699025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.698987    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.699025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.698997    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.700547 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.700535    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.700703 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.700680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.700741 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.700720    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.701273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.701257    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.701342 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.701288    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.701342 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.701298    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.701410 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.701257    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.701410 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.701363    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.701410 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.701372    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.702505 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.702492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.702547 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.702517    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:51.703211 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.703195    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:51.703289 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.703226    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:51.703289 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.703240    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:51.726239 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.726221    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-141.ec2.internal\" not found" node="ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.730628 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.730615    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-141.ec2.internal\" not found" node="ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.759121 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.759095    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:51.770638 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.770622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d5284f6e46d27b25e953ba9be3337b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal\" (UID: \"6d5284f6e46d27b25e953ba9be3337b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.770688 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.770644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5284f6e46d27b25e953ba9be3337b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal\" (UID: \"6d5284f6e46d27b25e953ba9be3337b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.770688 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.770664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26cb425dc070dd73c901ca637ab653e6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-141.ec2.internal\" (UID: \"26cb425dc070dd73c901ca637ab653e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.860192 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.860170    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:51.871084 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.871063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d5284f6e46d27b25e953ba9be3337b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal\" (UID: \"6d5284f6e46d27b25e953ba9be3337b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.871143 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.871089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5284f6e46d27b25e953ba9be3337b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal\" (UID: \"6d5284f6e46d27b25e953ba9be3337b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.871143 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.871109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26cb425dc070dd73c901ca637ab653e6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-141.ec2.internal\" (UID: \"26cb425dc070dd73c901ca637ab653e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.871214 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.871153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5284f6e46d27b25e953ba9be3337b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal\" (UID: \"6d5284f6e46d27b25e953ba9be3337b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.871214 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.871165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d5284f6e46d27b25e953ba9be3337b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal\" (UID: \"6d5284f6e46d27b25e953ba9be3337b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.871281 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:51.871220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26cb425dc070dd73c901ca637ab653e6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-141.ec2.internal\" (UID: \"26cb425dc070dd73c901ca637ab653e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:51.960523 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:51.960451    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.029925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.029891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:52.034168 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.033926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:52.060982 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:52.060958    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.161539 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:52.161508    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.262047 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:52.261976    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.362528 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:52.362492    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.374878 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.374856    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 08:20:52.375033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.375012    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:52.375077 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.375012    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:52.463311 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:52.463284    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.465459 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.465430    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:51 +0000 UTC" deadline="2028-06-12 00:10:43.141192866 +0000 UTC"
Sep 19 08:20:52.465546 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.465461    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15159h49m50.675738087s"
Sep 19 08:20:52.559320 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:52.559290    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5284f6e46d27b25e953ba9be3337b0.slice/crio-3306a8b2120f11dd4198091e6b43aa1b01473be137e2c0a0073c2bcd3c9816fa WatchSource:0}: Error finding container 3306a8b2120f11dd4198091e6b43aa1b01473be137e2c0a0073c2bcd3c9816fa: Status 404 returned error can't find the container with id 3306a8b2120f11dd4198091e6b43aa1b01473be137e2c0a0073c2bcd3c9816fa
Sep 19 08:20:52.559842 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:52.559823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cb425dc070dd73c901ca637ab653e6.slice/crio-663a26e444895399e2ca94229958f48ab6ad4c50ca7d0ab3b11afc662d9e3279 WatchSource:0}: Error finding container 663a26e444895399e2ca94229958f48ab6ad4c50ca7d0ab3b11afc662d9e3279: Status 404 returned error can't find the container with id 663a26e444895399e2ca94229958f48ab6ad4c50ca7d0ab3b11afc662d9e3279
Sep 19 08:20:52.563987 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:52.563965    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-141.ec2.internal\" not found"
Sep 19 08:20:52.566650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.566633    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:20:52.600527 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.600488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal" event={"ID":"26cb425dc070dd73c901ca637ab653e6","Type":"ContainerStarted","Data":"663a26e444895399e2ca94229958f48ab6ad4c50ca7d0ab3b11afc662d9e3279"}
Sep 19 08:20:52.601576 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.601553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal" event={"ID":"6d5284f6e46d27b25e953ba9be3337b0","Type":"ContainerStarted","Data":"3306a8b2120f11dd4198091e6b43aa1b01473be137e2c0a0073c2bcd3c9816fa"}
Sep 19 08:20:52.633533 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.633512    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:52.668387 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.668370    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:52.680098 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.680081    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:52.681732 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.681720    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal"
Sep 19 08:20:52.690052 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.690032    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:52.837107 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:52.837029    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:53.413941 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.413913    2579 apiserver.go:52] "Watching apiserver"
Sep 19 08:20:53.419651 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.419626    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 08:20:53.420564 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.420527    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-x6bcm","kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm","openshift-image-registry/node-ca-vmpk4","openshift-multus/multus-fl2k7","openshift-multus/network-metrics-daemon-tz7f6","openshift-network-diagnostics/network-check-target-tqgz5","openshift-network-operator/iptables-alerter-qc9bm","openshift-ovn-kubernetes/ovnkube-node-bkpds","kube-system/global-pull-secret-syncer-bx8qb","kube-system/konnectivity-agent-pwsg6","openshift-cluster-node-tuning-operator/tuned-d7k5h","openshift-dns/node-resolver-wwjml","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal"]
Sep 19 08:20:53.422283 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.422227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:53.422371 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.422310    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:20:53.423742 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.423721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.424906 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.424887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.425849 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.425822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.426203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.426185    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.426287 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.426273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.426345 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.426325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtngz\""
Sep 19 08:20:53.426826 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.426807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 08:20:53.427252 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.426993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.427252 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.427046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.427252 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.427165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.427520 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.427501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74b6j\""
Sep 19 08:20:53.427707 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.427656    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.427848 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.427833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.428149 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.428128    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 08:20:53.428245 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.428207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:53.428306 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.428264    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:20:53.428306 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.428128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h8qhq\""
Sep 19 08:20:53.429227 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.429210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.429312 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.429211    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 08:20:53.429368 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.429211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6t8md\""
Sep 19 08:20:53.429572 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.429554    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.429632 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.429574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.429683 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.429578    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 08:20:53.430890 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.430871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.431849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6qqrt\""
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.431859    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.431951    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.432007    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.432028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.432199    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.432263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.432339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.432273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 08:20:53.433120 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.433103    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mwxz5\""
Sep 19 08:20:53.433208 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.433186    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 08:20:53.433390 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.433353    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 08:20:53.433589 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.433574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.433655 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.433639    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:20:53.434368 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.434350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.434458 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.434437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lnfmt\""
Sep 19 08:20:53.435447 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.434978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.435447 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.435142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.435869 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.435850    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 08:20:53.436574 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.436558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.437165 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.437147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-86gfp\""
Sep 19 08:20:53.437250 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.437194    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 08:20:53.437306 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.437280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 08:20:53.438950 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.438929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:53.439095 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.439074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpwjw\""
Sep 19 08:20:53.439180 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.439120    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:53.468719 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.468689    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:53.469886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.469870    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 08:20:53.480919 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.480896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-cnibin\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.481029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.480933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-netns\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.481029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.480950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-env-overrides\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.480968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-systemd\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.481029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add4b376-afa3-493c-aeb1-916bfeacbf89-tmp-dir\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24pn\" (UniqueName: \"kubernetes.io/projected/add4b376-afa3-493c-aeb1-916bfeacbf89-kube-api-access-s24pn\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-system-cni-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c88771f-d05a-4033-bbc7-7c52addcb195-konnectivity-ca\") pod \"konnectivity-agent-pwsg6\" (UID: \"5c88771f-d05a-4033-bbc7-7c52addcb195\") " pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysconfig\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysctl-conf\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpd82\" (UniqueName: \"kubernetes.io/projected/ff198034-e56f-44da-8427-9046ff196291-kube-api-access-xpd82\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:53.481236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-kubelet\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-var-lib-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvt95\" (UniqueName: \"kubernetes.io/projected/a314ace4-fd72-46a7-96e0-81e1bb51a464-kube-api-access-vvt95\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de943430-0a59-4ea7-b59e-93636d87c4a0-dbus\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-ovnkube-config\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9kk\" (UniqueName: \"kubernetes.io/projected/202713c0-1ac1-483e-8389-57e9d34c23e4-kube-api-access-9l9kk\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysctl-d\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a13f1d56-3edb-463f-bee7-5a0b2af6423b-iptables-alerter-script\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvdb\" (UniqueName: \"kubernetes.io/projected/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-kube-api-access-fmvdb\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-slash\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-modprobe-d\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-socket-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.481650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9mjz\" (UniqueName: \"kubernetes.io/projected/193907a3-f925-4deb-a7c5-4707d8ee6788-kube-api-access-r9mjz\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a6f6ec9-e47c-4957-b28b-14014d1af467-serviceca\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-systemd-units\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-log-socket\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c88771f-d05a-4033-bbc7-7c52addcb195-agent-certs\") pod \"konnectivity-agent-pwsg6\" (UID: \"5c88771f-d05a-4033-bbc7-7c52addcb195\") " pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-run\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-var-lib-kubelet\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-cni-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-conf-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-cni-netd\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/202713c0-1ac1-483e-8389-57e9d34c23e4-ovn-node-metrics-cert\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw22j\" (UniqueName: \"kubernetes.io/projected/0a6f6ec9-e47c-4957-b28b-14014d1af467-kube-api-access-hw22j\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-cni-bin\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.481988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-node-log\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-sys-fs\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-etc-kubernetes\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.482424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rfpb\" (UniqueName: \"kubernetes.io/projected/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-kube-api-access-7rfpb\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/add4b376-afa3-493c-aeb1-916bfeacbf89-hosts-file\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-cni-bin\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-ovnkube-script-lib\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-sys\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-host\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-tuned\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-k8s-cni-cncf-io\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-cni-multus\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-run-netns\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-ovn\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-multus-certs\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-kubernetes\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-device-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a13f1d56-3edb-463f-bee7-5a0b2af6423b-host-slash\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.483194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-cni-binary-copy\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-kubelet\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-daemon-config\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a314ace4-fd72-46a7-96e0-81e1bb51a464-tmp\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvlnc\" (UniqueName: \"kubernetes.io/projected/a13f1d56-3edb-463f-bee7-5a0b2af6423b-kube-api-access-gvlnc\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de943430-0a59-4ea7-b59e-93636d87c4a0-kubelet-config\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-systemd\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-etc-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-lib-modules\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-system-cni-dir\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-socket-dir-parent\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-hostroot\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.482944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.483037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-registration-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.484064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.483078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-cnibin\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.484836 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.483096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:53.484836 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.483127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a6f6ec9-e47c-4957-b28b-14014d1af467-host\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.495916 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.495891    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:53.560613 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.560582    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ckz6z"
Sep 19 08:20:53.569359 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.569332    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ckz6z"
Sep 19 08:20:53.583404 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-systemd-units\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583404 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-log-socket\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c88771f-d05a-4033-bbc7-7c52addcb195-agent-certs\") pod \"konnectivity-agent-pwsg6\" (UID: \"5c88771f-d05a-4033-bbc7-7c52addcb195\") " pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-run\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-log-socket\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-var-lib-kubelet\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-systemd-units\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-cni-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.583587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-run\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-conf-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-var-lib-kubelet\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-cni-netd\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-conf-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-cni-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/202713c0-1ac1-483e-8389-57e9d34c23e4-ovn-node-metrics-cert\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-cni-netd\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw22j\" (UniqueName: \"kubernetes.io/projected/0a6f6ec9-e47c-4957-b28b-14014d1af467-kube-api-access-hw22j\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-cni-bin\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-node-log\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583829    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-sys-fs\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-etc-kubernetes\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.583930 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-node-log\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfpb\" (UniqueName: \"kubernetes.io/projected/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-kube-api-access-7rfpb\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.583992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-etc-kubernetes\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.584020    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-sys-fs\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.584134    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:54.084083581 +0000 UTC m=+4.749219128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/add4b376-afa3-493c-aeb1-916bfeacbf89-hosts-file\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-cni-bin\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.584221    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-ovnkube-script-lib\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-sys\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-host\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/add4b376-afa3-493c-aeb1-916bfeacbf89-hosts-file\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-tuned\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-k8s-cni-cncf-io\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.584606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-cni-bin\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-cni-multus\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-sys\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.584405    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:54.08436972 +0000 UTC m=+4.749505258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-cni-multus\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-k8s-cni-cncf-io\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-host\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-run-netns\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-ovn\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-multus-certs\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-cni-bin\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-kubernetes\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-multus-certs\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-device-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-run-netns\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-kubernetes\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-ovn\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.585429 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a13f1d56-3edb-463f-bee7-5a0b2af6423b-host-slash\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-device-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-cni-binary-copy\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a13f1d56-3edb-463f-bee7-5a0b2af6423b-host-slash\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-kubelet\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-daemon-config\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-var-lib-kubelet\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-ovnkube-script-lib\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.584971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a314ace4-fd72-46a7-96e0-81e1bb51a464-tmp\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvlnc\" (UniqueName: \"kubernetes.io/projected/a13f1d56-3edb-463f-bee7-5a0b2af6423b-kube-api-access-gvlnc\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de943430-0a59-4ea7-b59e-93636d87c4a0-kubelet-config\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-systemd\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-etc-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-lib-modules\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-system-cni-dir\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.586265 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-socket-dir-parent\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-hostroot\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-cni-binary-copy\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-registration-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-cnibin\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-socket-dir-parent\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-lib-modules\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a6f6ec9-e47c-4957-b28b-14014d1af467-host\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-system-cni-dir\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a6f6ec9-e47c-4957-b28b-14014d1af467-host\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-cnibin\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-systemd\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-netns\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-cnibin\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de943430-0a59-4ea7-b59e-93636d87c4a0-kubelet-config\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.587091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-hostroot\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-etc-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-registration-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/193907a3-f925-4deb-a7c5-4707d8ee6788-cnibin\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-env-overrides\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-systemd\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add4b376-afa3-493c-aeb1-916bfeacbf89-tmp-dir\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-host-run-netns\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24pn\" (UniqueName: \"kubernetes.io/projected/add4b376-afa3-493c-aeb1-916bfeacbf89-kube-api-access-s24pn\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-systemd\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-system-cni-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c88771f-d05a-4033-bbc7-7c52addcb195-konnectivity-ca\") pod \"konnectivity-agent-pwsg6\" (UID: \"5c88771f-d05a-4033-bbc7-7c52addcb195\") " pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysconfig\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysctl-conf\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpd82\" (UniqueName: \"kubernetes.io/projected/ff198034-e56f-44da-8427-9046ff196291-kube-api-access-xpd82\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:53.587940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-kubelet\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-var-lib-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvt95\" (UniqueName: \"kubernetes.io/projected/a314ace4-fd72-46a7-96e0-81e1bb51a464-kube-api-access-vvt95\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.585997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de943430-0a59-4ea7-b59e-93636d87c4a0-dbus\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-ovnkube-config\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9kk\" (UniqueName: \"kubernetes.io/projected/202713c0-1ac1-483e-8389-57e9d34c23e4-kube-api-access-9l9kk\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysctl-d\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a13f1d56-3edb-463f-bee7-5a0b2af6423b-iptables-alerter-script\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvdb\" (UniqueName: \"kubernetes.io/projected/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-kube-api-access-fmvdb\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add4b376-afa3-493c-aeb1-916bfeacbf89-tmp-dir\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-slash\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-slash\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-env-overrides\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysconfig\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.588577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.586968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysctl-conf\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-system-cni-dir\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-kubelet\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-var-lib-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-modprobe-d\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-socket-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-run-openvswitch\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/193907a3-f925-4deb-a7c5-4707d8ee6788-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9mjz\" (UniqueName: \"kubernetes.io/projected/193907a3-f925-4deb-a7c5-4707d8ee6788-kube-api-access-r9mjz\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a6f6ec9-e47c-4957-b28b-14014d1af467-serviceca\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de943430-0a59-4ea7-b59e-93636d87c4a0-dbus\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a314ace4-fd72-46a7-96e0-81e1bb51a464-tmp\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c88771f-d05a-4033-bbc7-7c52addcb195-konnectivity-ca\") pod \"konnectivity-agent-pwsg6\" (UID: \"5c88771f-d05a-4033-bbc7-7c52addcb195\") " pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.589356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/202713c0-1ac1-483e-8389-57e9d34c23e4-ovn-node-metrics-cert\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-modprobe-d\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-sysctl-d\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202713c0-1ac1-483e-8389-57e9d34c23e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a6f6ec9-e47c-4957-b28b-14014d1af467-serviceca\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c88771f-d05a-4033-bbc7-7c52addcb195-agent-certs\") pod \"konnectivity-agent-pwsg6\" (UID: \"5c88771f-d05a-4033-bbc7-7c52addcb195\") " pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587950    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-socket-dir\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.587953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/202713c0-1ac1-483e-8389-57e9d34c23e4-ovnkube-config\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.588180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a13f1d56-3edb-463f-bee7-5a0b2af6423b-iptables-alerter-script\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.590146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.588386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-multus-daemon-config\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.590964 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.590946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a314ace4-fd72-46a7-96e0-81e1bb51a464-etc-tuned\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.599628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfpb\" (UniqueName: \"kubernetes.io/projected/643b96e5-3186-43f1-ba3f-b4bcf006c0ea-kube-api-access-7rfpb\") pod \"aws-ebs-csi-driver-node-hw4gm\" (UID: \"643b96e5-3186-43f1-ba3f-b4bcf006c0ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.600427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9kk\" (UniqueName: \"kubernetes.io/projected/202713c0-1ac1-483e-8389-57e9d34c23e4-kube-api-access-9l9kk\") pod \"ovnkube-node-bkpds\" (UID: \"202713c0-1ac1-483e-8389-57e9d34c23e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.600525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24pn\" (UniqueName: \"kubernetes.io/projected/add4b376-afa3-493c-aeb1-916bfeacbf89-kube-api-access-s24pn\") pod \"node-resolver-wwjml\" (UID: \"add4b376-afa3-493c-aeb1-916bfeacbf89\") " pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.600542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw22j\" (UniqueName: \"kubernetes.io/projected/0a6f6ec9-e47c-4957-b28b-14014d1af467-kube-api-access-hw22j\") pod \"node-ca-vmpk4\" (UID: \"0a6f6ec9-e47c-4957-b28b-14014d1af467\") " pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.600620    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.600637    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.600650    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:53.600734    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:54.100713641 +0000 UTC m=+4.765849186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.600911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9mjz\" (UniqueName: \"kubernetes.io/projected/193907a3-f925-4deb-a7c5-4707d8ee6788-kube-api-access-r9mjz\") pod \"multus-additional-cni-plugins-x6bcm\" (UID: \"193907a3-f925-4deb-a7c5-4707d8ee6788\") " pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.602717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.601606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpd82\" (UniqueName: \"kubernetes.io/projected/ff198034-e56f-44da-8427-9046ff196291-kube-api-access-xpd82\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:53.604178 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.604120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvlnc\" (UniqueName: \"kubernetes.io/projected/a13f1d56-3edb-463f-bee7-5a0b2af6423b-kube-api-access-gvlnc\") pod \"iptables-alerter-qc9bm\" (UID: \"a13f1d56-3edb-463f-bee7-5a0b2af6423b\") " pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.604852 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.604827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvdb\" (UniqueName: \"kubernetes.io/projected/e8a9c0f8-f161-40df-8ad5-7b6059dd1baa-kube-api-access-fmvdb\") pod \"multus-fl2k7\" (UID: \"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa\") " pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.606384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.606366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvt95\" (UniqueName: \"kubernetes.io/projected/a314ace4-fd72-46a7-96e0-81e1bb51a464-kube-api-access-vvt95\") pod \"tuned-d7k5h\" (UID: \"a314ace4-fd72-46a7-96e0-81e1bb51a464\") " pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.720543 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.720476    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:53.736488 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.736441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwjml"
Sep 19 08:20:53.743410 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.743391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm"
Sep 19 08:20:53.750000 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.749983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vmpk4"
Sep 19 08:20:53.758246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.758226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fl2k7"
Sep 19 08:20:53.763294 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.763277    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:20:53.770541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.770519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6bcm"
Sep 19 08:20:53.777591 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.777574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qc9bm"
Sep 19 08:20:53.783875 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.783857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:20:53.789528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.789510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7k5h"
Sep 19 08:20:53.867037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:53.867005    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:54.089975 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.089904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:54.089975 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.089949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:54.090165 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.090049    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:54.090165 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.090100    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:54.090165 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.090115    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:55.09009352 +0000 UTC m=+5.755229049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:54.090165 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.090157    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:55.090132506 +0000 UTC m=+5.755268048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:54.190661 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.190631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:54.190853 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.190841    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:54.190900 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.190862    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:54.190900 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.190873    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:54.190987 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.190934    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:55.190920479 +0000 UTC m=+5.856056003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:54.331308 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.331119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202713c0_1ac1_483e_8389_57e9d34c23e4.slice/crio-b39a8e66286379d8f7d659fac09adb124e93f808e59732d6a87a621cc0c6ae3c WatchSource:0}: Error finding container b39a8e66286379d8f7d659fac09adb124e93f808e59732d6a87a621cc0c6ae3c: Status 404 returned error can't find the container with id b39a8e66286379d8f7d659fac09adb124e93f808e59732d6a87a621cc0c6ae3c
Sep 19 08:20:54.338595 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.338566    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193907a3_f925_4deb_a7c5_4707d8ee6788.slice/crio-3dbd3258ef2327b15829aa5b51b822cd33447d59c1ea4f941e8475dd92ee5b0a WatchSource:0}: Error finding container 3dbd3258ef2327b15829aa5b51b822cd33447d59c1ea4f941e8475dd92ee5b0a: Status 404 returned error can't find the container with id 3dbd3258ef2327b15829aa5b51b822cd33447d59c1ea4f941e8475dd92ee5b0a
Sep 19 08:20:54.339233 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.339166    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c88771f_d05a_4033_bbc7_7c52addcb195.slice/crio-e2e1e94943ebdb4b99222ba575abdeb8cb3468a0f9a78d53c92c7bc532e51b9f WatchSource:0}: Error finding container e2e1e94943ebdb4b99222ba575abdeb8cb3468a0f9a78d53c92c7bc532e51b9f: Status 404 returned error can't find the container with id e2e1e94943ebdb4b99222ba575abdeb8cb3468a0f9a78d53c92c7bc532e51b9f
Sep 19 08:20:54.340111 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.340048    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda314ace4_fd72_46a7_96e0_81e1bb51a464.slice/crio-a7e0e6abfcee176ac3c9fb7a38684ba78090dcecf89a5192a186e9662a086bff WatchSource:0}: Error finding container a7e0e6abfcee176ac3c9fb7a38684ba78090dcecf89a5192a186e9662a086bff: Status 404 returned error can't find the container with id a7e0e6abfcee176ac3c9fb7a38684ba78090dcecf89a5192a186e9662a086bff
Sep 19 08:20:54.342208 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.342180    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a9c0f8_f161_40df_8ad5_7b6059dd1baa.slice/crio-a590453de84c89d23ce1e998b753a08d642175563d2743ce32b939d9cee94b9a WatchSource:0}: Error finding container a590453de84c89d23ce1e998b753a08d642175563d2743ce32b939d9cee94b9a: Status 404 returned error can't find the container with id a590453de84c89d23ce1e998b753a08d642175563d2743ce32b939d9cee94b9a
Sep 19 08:20:54.343651 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.343631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13f1d56_3edb_463f_bee7_5a0b2af6423b.slice/crio-132d61d4bcbe2aec0f91d360992557ff355260d3467d44318c866f73b9176ab1 WatchSource:0}: Error finding container 132d61d4bcbe2aec0f91d360992557ff355260d3467d44318c866f73b9176ab1: Status 404 returned error can't find the container with id 132d61d4bcbe2aec0f91d360992557ff355260d3467d44318c866f73b9176ab1
Sep 19 08:20:54.344387 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:20:54.344367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6f6ec9_e47c_4957_b28b_14014d1af467.slice/crio-1032798fbc43e9c79f70f1da12c7bfe4f325151496974fc6c146e84832479f0e WatchSource:0}: Error finding container 1032798fbc43e9c79f70f1da12c7bfe4f325151496974fc6c146e84832479f0e: Status 404 returned error can't find the container with id 1032798fbc43e9c79f70f1da12c7bfe4f325151496974fc6c146e84832479f0e
Sep 19 08:20:54.571675 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.571644    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:53 +0000 UTC" deadline="2028-07-04 22:49:47.000798568 +0000 UTC"
Sep 19 08:20:54.571675 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.571670    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15710h28m52.429131471s"
Sep 19 08:20:54.597989 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.597921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:54.598076 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.598032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:20:54.598122 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.598084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:54.598215 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:54.598194    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:20:54.605775 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.605751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal" event={"ID":"26cb425dc070dd73c901ca637ab653e6","Type":"ContainerStarted","Data":"6dd5b8b84584ab6433d11350175f464d197c1e25312d033f173ea8a92ccdd151"}
Sep 19 08:20:54.606852 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.606833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vmpk4" event={"ID":"0a6f6ec9-e47c-4957-b28b-14014d1af467","Type":"ContainerStarted","Data":"1032798fbc43e9c79f70f1da12c7bfe4f325151496974fc6c146e84832479f0e"}
Sep 19 08:20:54.607792 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.607774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fl2k7" event={"ID":"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa","Type":"ContainerStarted","Data":"a590453de84c89d23ce1e998b753a08d642175563d2743ce32b939d9cee94b9a"}
Sep 19 08:20:54.608822 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.608801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm" event={"ID":"643b96e5-3186-43f1-ba3f-b4bcf006c0ea","Type":"ContainerStarted","Data":"86dbda1c32b526c3663e2f4ef9612fab232eef7570ad6bfb0ecafcbdd1f84c49"}
Sep 19 08:20:54.610016 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.609993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwjml" event={"ID":"add4b376-afa3-493c-aeb1-916bfeacbf89","Type":"ContainerStarted","Data":"8e91b2fec690a5135c0e3d3e730ef406617de977f793abc1a739704ec62d4d12"}
Sep 19 08:20:54.611380 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.611358    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d5284f6e46d27b25e953ba9be3337b0" containerID="9c2bf113885c86edcb911dd79d871215ea35ddae2ce1643313e87d2e48a06687" exitCode=0
Sep 19 08:20:54.611458 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.611425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal" event={"ID":"6d5284f6e46d27b25e953ba9be3337b0","Type":"ContainerDied","Data":"9c2bf113885c86edcb911dd79d871215ea35ddae2ce1643313e87d2e48a06687"}
Sep 19 08:20:54.612818 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.612797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qc9bm" event={"ID":"a13f1d56-3edb-463f-bee7-5a0b2af6423b","Type":"ContainerStarted","Data":"132d61d4bcbe2aec0f91d360992557ff355260d3467d44318c866f73b9176ab1"}
Sep 19 08:20:54.613877 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.613858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pwsg6" event={"ID":"5c88771f-d05a-4033-bbc7-7c52addcb195","Type":"ContainerStarted","Data":"e2e1e94943ebdb4b99222ba575abdeb8cb3468a0f9a78d53c92c7bc532e51b9f"}
Sep 19 08:20:54.614948 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.614929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7k5h" event={"ID":"a314ace4-fd72-46a7-96e0-81e1bb51a464","Type":"ContainerStarted","Data":"a7e0e6abfcee176ac3c9fb7a38684ba78090dcecf89a5192a186e9662a086bff"}
Sep 19 08:20:54.615994 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.615969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerStarted","Data":"3dbd3258ef2327b15829aa5b51b822cd33447d59c1ea4f941e8475dd92ee5b0a"}
Sep 19 08:20:54.616932 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.616915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"b39a8e66286379d8f7d659fac09adb124e93f808e59732d6a87a621cc0c6ae3c"}
Sep 19 08:20:54.620552 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:54.620507    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-141.ec2.internal" podStartSLOduration=2.6204958879999998 podStartE2EDuration="2.620495888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:54.619820856 +0000 UTC m=+5.284956404" watchObservedRunningTime="2026-09-19 08:20:54.620495888 +0000 UTC m=+5.285631434"
Sep 19 08:20:55.098053 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.097946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:55.098053 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.098041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:55.098285 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.098233    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:55.098338 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.098300    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:57.098279697 +0000 UTC m=+7.763415228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:55.098715 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.098674    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:55.098816 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.098744    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:57.098730794 +0000 UTC m=+7.763866322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:55.199108 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.199063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:55.199291 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.199268    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:55.199291 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.199288    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:55.199399 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.199300    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:55.199399 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.199362    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:57.199342968 +0000 UTC m=+7.864478517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:55.572730 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.572593    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:53 +0000 UTC" deadline="2028-07-02 16:01:11.642713934 +0000 UTC"
Sep 19 08:20:55.572730 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.572651    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15655h40m16.070067177s"
Sep 19 08:20:55.598825 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.598326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:55.598825 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:55.598469    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:20:55.641802 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:55.641762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal" event={"ID":"6d5284f6e46d27b25e953ba9be3337b0","Type":"ContainerStarted","Data":"d67f05e6e903b4a1790325d08fa4191f373b240e0a4c5b7fe7f1c40401e02785"}
Sep 19 08:20:56.598223 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:56.598189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:56.598743 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:56.598318    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:20:56.598743 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:56.598716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:56.598873 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:56.598806    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:20:57.118837 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:57.118798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:57.119025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:57.118851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:57.119127 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.119094    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:57.119186 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.119165    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:01.119146537 +0000 UTC m=+11.784282084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:57.119570 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.119553    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:57.119649 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.119607    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:01.119593291 +0000 UTC m=+11.784728827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:57.219775 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:57.219712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:57.219999 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.219965    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:57.219999 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.219991    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:57.220214 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.220004    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:57.220214 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.220065    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:01.220045099 +0000 UTC m=+11.885180638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:57.599390 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:57.599308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:57.599844 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:57.599456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:20:58.598790 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:58.598737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:20:58.598978 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:58.598878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:20:58.599271 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:58.599240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:20:58.599369 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:58.599330    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:20:59.598230 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:20:59.598156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:20:59.598583 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:20:59.598278    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:00.598183 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:00.598122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:00.598365 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:00.598235    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:00.598717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:00.598563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:00.598717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:00.598636    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:01.155324 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:01.155292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:01.155524 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:01.155338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:01.155524 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.155456    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:01.155524 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.155487    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:01.155707 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.155535    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.155513886 +0000 UTC m=+19.820649422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:01.155707 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.155557    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.155545739 +0000 UTC m=+19.820681267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:01.256007 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:01.255959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:01.256173 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.256142    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:01.256173 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.256164    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:01.256284 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.256176    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:01.256284 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.256218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.256206034 +0000 UTC m=+19.921341558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:01.599443 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:01.599366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:01.599837 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:01.599498    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:02.598582 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:02.598551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:02.598750 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:02.598605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:02.598750 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:02.598727    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:02.598994 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:02.598963    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:03.598837 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:03.598804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:03.599266 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:03.598933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:04.598223 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:04.598183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:04.598223 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:04.598208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:04.598509 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:04.598327    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:04.598509 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:04.598438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:05.598435 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:05.598395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:05.598935 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:05.598537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:06.598437 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:06.598405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:06.598615 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:06.598405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:06.598615 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:06.598519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:06.598615 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:06.598604    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:07.598400 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:07.598367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:07.598580 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:07.598503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:08.598426 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:08.598395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:08.599024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:08.598401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:08.599024 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:08.598503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:08.599024 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:08.598593    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:09.207412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:09.207370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:09.207412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:09.207421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:09.207725 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.207545    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:09.207725 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.207546    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:09.207725 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.207601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.207583296 +0000 UTC m=+35.872718819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:09.207725 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.207618    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.207609499 +0000 UTC m=+35.872745031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:09.308118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:09.308084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:09.308296 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.308230    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:09.308296 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.308249    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:09.308296 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.308259    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:09.308410 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.308316    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.30829977 +0000 UTC m=+35.973435314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:09.598396 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:09.598363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:09.598572 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:09.598495    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:10.598789 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:10.598753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:10.599246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:10.598753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:10.599246 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:10.598887    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:10.599246 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:10.598941    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:11.600015 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:11.599622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:11.600015 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:11.599908    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:11.682054 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:11.681570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm" event={"ID":"643b96e5-3186-43f1-ba3f-b4bcf006c0ea","Type":"ContainerStarted","Data":"5d50d5422abca364bd49eb79f3df6fbd23864e7241a3bdb9bdf8068a391af809"}
Sep 19 08:21:11.683412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:11.683387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pwsg6" event={"ID":"5c88771f-d05a-4033-bbc7-7c52addcb195","Type":"ContainerStarted","Data":"4767024fd1b769007012831759332205bcb2cdc49e324444137486b4a4f66275"}
Sep 19 08:21:11.685865 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:11.685840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7k5h" event={"ID":"a314ace4-fd72-46a7-96e0-81e1bb51a464","Type":"ContainerStarted","Data":"04102c4eacddf1658cc2dda2cb7e817f90f7410ab9e68c9c2b0b53228521c5f5"}
Sep 19 08:21:11.703263 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:11.702444    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-141.ec2.internal" podStartSLOduration=19.70242777 podStartE2EDuration="19.70242777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.658301259 +0000 UTC m=+6.323436806" watchObservedRunningTime="2026-09-19 08:21:11.70242777 +0000 UTC m=+22.367563320"
Sep 19 08:21:12.598163 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.597904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:12.598272 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:12.598206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:12.598272 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.598243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:12.598373 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:12.598335    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:12.689462 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.689436    2579 generic.go:356] "Generic (PLEG): container finished" podID="193907a3-f925-4deb-a7c5-4707d8ee6788" containerID="65b291234f42807a8d1b2e6b49dcd8ae2af02a104618f0edf6c1349598fe726e" exitCode=0
Sep 19 08:21:12.690106 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.689519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerDied","Data":"65b291234f42807a8d1b2e6b49dcd8ae2af02a104618f0edf6c1349598fe726e"}
Sep 19 08:21:12.692621 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.692605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:21:12.693021 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.692994    2579 generic.go:356] "Generic (PLEG): container finished" podID="202713c0-1ac1-483e-8389-57e9d34c23e4" containerID="66bba5a5b4b6d96387304a21bb3eab94620272e34b2c2754fe29f533e0404784" exitCode=1
Sep 19 08:21:12.693113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.693061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"f013abf83bc9923d9f22c847e3077bd1e4fde58d29ea1a5c7b144633935f8677"}
Sep 19 08:21:12.693163 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.693112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"b5a16f9fe5eab4339d2f59b98cbec3b977ddb419c6729159b916eab5825e168f"}
Sep 19 08:21:12.693163 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.693157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"3098b253992ea0fa9a3f1b3da3f0798ce367ec04e58cb2f9ce17dafd9f4209f1"}
Sep 19 08:21:12.693249 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.693206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"5da6fdf8389ed9ffeef633eda5d7aca3fa9de716de3c78af7d563f02c8f85e51"}
Sep 19 08:21:12.693281 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.693255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerDied","Data":"66bba5a5b4b6d96387304a21bb3eab94620272e34b2c2754fe29f533e0404784"}
Sep 19 08:21:12.693319 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.693309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"dadd509822798cc036743121fec96e296be05a9617fbf2505677e4df65c5b08d"}
Sep 19 08:21:12.694739 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.694715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vmpk4" event={"ID":"0a6f6ec9-e47c-4957-b28b-14014d1af467","Type":"ContainerStarted","Data":"a70004b6c26e896a5cf1be1fb8594e5dc308436640f06d7981c14bfdb3b1628f"}
Sep 19 08:21:12.696411 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.696388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fl2k7" event={"ID":"e8a9c0f8-f161-40df-8ad5-7b6059dd1baa","Type":"ContainerStarted","Data":"d5c4a630a723abdefaa6cb4c36d35344be7f71179d0a96d805ded1f3ac416a1c"}
Sep 19 08:21:12.698019 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.697937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwjml" event={"ID":"add4b376-afa3-493c-aeb1-916bfeacbf89","Type":"ContainerStarted","Data":"e098719b2c3982a7f7759787354cdcbaf7f5f97fe8d49284a83c023f8cf65835"}
Sep 19 08:21:12.705784 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.705767    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 08:21:12.713972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.713840    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pwsg6" podStartSLOduration=4.611383176 podStartE2EDuration="21.713823714s" totalImagesPullingTime="17.102440538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.703154477 +0000 UTC m=+22.368290025" watchObservedRunningTime="2026-09-19 08:21:12.713823714 +0000 UTC m=+23.378959262"
Sep 19 08:21:12.730536 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.730497    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wwjml" podStartSLOduration=4.612825195 podStartE2EDuration="21.730482793s" totalImagesPullingTime="17.117657598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.730385053 +0000 UTC m=+23.395520600" watchObservedRunningTime="2026-09-19 08:21:12.730482793 +0000 UTC m=+23.395618338"
Sep 19 08:21:12.745587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.745555    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vmpk4" podStartSLOduration=4.627981417 podStartE2EDuration="21.745544144s" totalImagesPullingTime="17.117562727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.745431261 +0000 UTC m=+23.410566806" watchObservedRunningTime="2026-09-19 08:21:12.745544144 +0000 UTC m=+23.410679690"
Sep 19 08:21:12.787342 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:12.787304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fl2k7" podStartSLOduration=4.637642371 podStartE2EDuration="21.787293337s" totalImagesPullingTime="17.149650966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.765240231 +0000 UTC m=+23.430375776" watchObservedRunningTime="2026-09-19 08:21:12.787293337 +0000 UTC m=+23.452428883"
Sep 19 08:21:13.547970 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.547860    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:21:12.7057819Z","UUID":"acd8883a-3d4b-40d2-a3e7-eca558d3ae6e","Handler":null,"Name":"","Endpoint":""}
Sep 19 08:21:13.549430 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.549409    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 08:21:13.549534 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.549457    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 08:21:13.598688 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.598661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:13.598838 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:13.598814    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:13.702520 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.702455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm" event={"ID":"643b96e5-3186-43f1-ba3f-b4bcf006c0ea","Type":"ContainerStarted","Data":"357c695e309f26f72dea69d8421a88619a27aaf50b5bbf4ce9d5aaef6360c171"}
Sep 19 08:21:13.704075 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.704046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qc9bm" event={"ID":"a13f1d56-3edb-463f-bee7-5a0b2af6423b","Type":"ContainerStarted","Data":"3d06d9da8f56e4c01350e02445dc8c92757ac052efe50ac67833c2af3c61bf8f"}
Sep 19 08:21:13.722230 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.722184    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qc9bm" podStartSLOduration=5.596747404 podStartE2EDuration="22.722166103s" totalImagesPullingTime="17.125418699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:13.721965677 +0000 UTC m=+24.387101243" watchObservedRunningTime="2026-09-19 08:21:13.722166103 +0000 UTC m=+24.387301649"
Sep 19 08:21:13.722857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:13.722516    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d7k5h" podStartSLOduration=5.609224864 podStartE2EDuration="22.7225077s" totalImagesPullingTime="17.113282836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.787890076 +0000 UTC m=+23.453025644" watchObservedRunningTime="2026-09-19 08:21:13.7225077 +0000 UTC m=+24.387643247"
Sep 19 08:21:14.598973 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:14.598747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:14.599115 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:14.598780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:14.599115 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:14.599013    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:14.599205 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:14.599129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:14.708554 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:14.708521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm" event={"ID":"643b96e5-3186-43f1-ba3f-b4bcf006c0ea","Type":"ContainerStarted","Data":"9c66c32138c744e169424fbc3a9f2feb4e2e9eab050ae9724f2a801d570f31d6"}
Sep 19 08:21:14.712167 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:14.712142    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:21:14.712556 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:14.712530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"2349195570d1fb5df97908ff830f1c28a146d3c7ae4b850aa587884734fb52f8"}
Sep 19 08:21:14.727558 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:14.727509    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hw4gm" podStartSLOduration=4.7072690569999995 podStartE2EDuration="23.727490798s" totalImagesPullingTime="19.020221741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:14.726441207 +0000 UTC m=+25.391576752" watchObservedRunningTime="2026-09-19 08:21:14.727490798 +0000 UTC m=+25.392626344"
Sep 19 08:21:15.350557 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:15.350525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:21:15.351584 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:15.351566    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:21:15.597809 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:15.597772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:15.597974 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:15.597898    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:15.715600 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:15.715529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:21:15.716220 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:15.715900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pwsg6"
Sep 19 08:21:16.598472 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.598451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:16.598555 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:16.598534    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:16.598633 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.598617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:16.598749 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:16.598732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:16.720589 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.720260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerStarted","Data":"c1037ea6b16ff6c98861251608869ffc7974e3f65af97e8dee82ec9c47ad78b4"}
Sep 19 08:21:16.724369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.724344    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:21:16.725303 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.725281    2579 scope.go:117] "RemoveContainer" containerID="66bba5a5b4b6d96387304a21bb3eab94620272e34b2c2754fe29f533e0404784"
Sep 19 08:21:16.725570 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.725493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"4646127a176f281f41ca41e4660bea221976b92f1e6357d64ff59f4c39b21709"}
Sep 19 08:21:16.725684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.725596    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:21:16.725684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.725649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:21:16.725811 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.725722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:21:16.749611 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.749587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:21:16.751260 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:16.751236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:21:17.598557 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:17.598531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:17.598678 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:17.598658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:17.728502 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:17.728472    2579 generic.go:356] "Generic (PLEG): container finished" podID="193907a3-f925-4deb-a7c5-4707d8ee6788" containerID="c1037ea6b16ff6c98861251608869ffc7974e3f65af97e8dee82ec9c47ad78b4" exitCode=0
Sep 19 08:21:17.728913 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:17.728556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerDied","Data":"c1037ea6b16ff6c98861251608869ffc7974e3f65af97e8dee82ec9c47ad78b4"}
Sep 19 08:21:17.732325 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:17.732305    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:21:17.732678 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:17.732656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" event={"ID":"202713c0-1ac1-483e-8389-57e9d34c23e4","Type":"ContainerStarted","Data":"0d9a201442f25be47894595c50e6627b6cd2500fe582e1815f596e439964d255"}
Sep 19 08:21:17.775921 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:17.775880    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds" podStartSLOduration=9.630563284 podStartE2EDuration="26.775866939s" totalImagesPullingTime="17.145303655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:17.77522877 +0000 UTC m=+28.440364306" watchObservedRunningTime="2026-09-19 08:21:17.775866939 +0000 UTC m=+28.441002485"
Sep 19 08:21:18.506903 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.506730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqgz5"]
Sep 19 08:21:18.507028 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.507016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:18.507129 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:18.507105    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:18.509964 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.509938    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx8qb"]
Sep 19 08:21:18.510082 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.510057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:18.510181 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:18.510161    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:18.510710 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.510666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tz7f6"]
Sep 19 08:21:18.510829 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.510813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:18.510966 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:18.510944    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:18.736648 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.736616    2579 generic.go:356] "Generic (PLEG): container finished" podID="193907a3-f925-4deb-a7c5-4707d8ee6788" containerID="24eca81e3f6d55f1630545a6de023f2f7642321e43798596d160b86c3a2e3c7a" exitCode=0
Sep 19 08:21:18.737068 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:18.736708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerDied","Data":"24eca81e3f6d55f1630545a6de023f2f7642321e43798596d160b86c3a2e3c7a"}
Sep 19 08:21:19.741170 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:19.741140    2579 generic.go:356] "Generic (PLEG): container finished" podID="193907a3-f925-4deb-a7c5-4707d8ee6788" containerID="4656dda6ee95ff4ab3b35c8976a3809b96cdc994fd4e564d349cd1a93d8fdcc2" exitCode=0
Sep 19 08:21:19.741491 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:19.741208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerDied","Data":"4656dda6ee95ff4ab3b35c8976a3809b96cdc994fd4e564d349cd1a93d8fdcc2"}
Sep 19 08:21:20.598215 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:20.598147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:20.598329 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:20.598263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:20.598329 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:20.598279    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:20.598329 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:20.598308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:20.598450 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:20.598398    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:20.598491 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:20.598471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:22.597977 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:22.597939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:22.597977 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:22.597978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:22.598717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:22.597985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:22.598717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:22.598063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:22.598717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:22.598157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:22.598717 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:22.598231    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:23.757320 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:23.757268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerStarted","Data":"3af2d5577e10c87979579d2c15b3e623223f43c0ab25ed4bd0df8d4c6ef68457"}
Sep 19 08:21:24.598472 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.598445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:24.598564 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.598445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:24.598564 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:24.598550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tz7f6" podUID="ff198034-e56f-44da-8427-9046ff196291"
Sep 19 08:21:24.598633 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.598558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:24.598633 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:24.598608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx8qb" podUID="de943430-0a59-4ea7-b59e-93636d87c4a0"
Sep 19 08:21:24.598781 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:24.598760    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgz5" podUID="eadec7b8-90a1-465f-8731-eed5d4bdfd49"
Sep 19 08:21:24.704264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.704244    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-141.ec2.internal" event="NodeReady"
Sep 19 08:21:24.704361 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.704326    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 08:21:24.762381 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.762356    2579 generic.go:356] "Generic (PLEG): container finished" podID="193907a3-f925-4deb-a7c5-4707d8ee6788" containerID="3af2d5577e10c87979579d2c15b3e623223f43c0ab25ed4bd0df8d4c6ef68457" exitCode=0
Sep 19 08:21:24.762713 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.762415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerDied","Data":"3af2d5577e10c87979579d2c15b3e623223f43c0ab25ed4bd0df8d4c6ef68457"}
Sep 19 08:21:24.765373 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.765328    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-84qvd"]
Sep 19 08:21:24.768263 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.768248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:24.769390 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.769370    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9zgzq"]
Sep 19 08:21:24.770915 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.770896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:24.772251 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.772236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 08:21:24.772673 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.772659    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 08:21:24.772781 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.772715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g2bvg\""
Sep 19 08:21:24.772841 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.772817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 08:21:24.772988 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.772964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 08:21:24.773213 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.773193    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 08:21:24.773293 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.773276    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vbr8d\""
Sep 19 08:21:24.787369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.787186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9zgzq"]
Sep 19 08:21:24.788113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.788043    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-84qvd"]
Sep 19 08:21:24.925384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.925356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkzc\" (UniqueName: \"kubernetes.io/projected/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-kube-api-access-9pkzc\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:24.925528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.925399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/991799e8-945e-4c15-aeaf-a664b5f10700-cert\") pod \"ingress-canary-9zgzq\" (UID: \"991799e8-945e-4c15-aeaf-a664b5f10700\") " pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:24.925528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.925426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:24.925528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.925450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-tmp-dir\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:24.925684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.925562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-config-volume\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:24.925684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:24.925629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkj4\" (UniqueName: \"kubernetes.io/projected/991799e8-945e-4c15-aeaf-a664b5f10700-kube-api-access-bfkj4\") pod \"ingress-canary-9zgzq\" (UID: \"991799e8-945e-4c15-aeaf-a664b5f10700\") " pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:25.026654 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.026627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-config-volume\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.026757 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.026672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkj4\" (UniqueName: \"kubernetes.io/projected/991799e8-945e-4c15-aeaf-a664b5f10700-kube-api-access-bfkj4\") pod \"ingress-canary-9zgzq\" (UID: \"991799e8-945e-4c15-aeaf-a664b5f10700\") " pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:25.026820 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.026804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkzc\" (UniqueName: \"kubernetes.io/projected/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-kube-api-access-9pkzc\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.026855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.026844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/991799e8-945e-4c15-aeaf-a664b5f10700-cert\") pod \"ingress-canary-9zgzq\" (UID: \"991799e8-945e-4c15-aeaf-a664b5f10700\") " pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:25.026899 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.026861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.026899 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.026876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-tmp-dir\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.027014 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.026971    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.027066 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.027034    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls podName:5c35a6f7-ada6-4695-a9d8-6e3d8560790a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.527016457 +0000 UTC m=+36.192151995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls") pod "dns-default-84qvd" (UID: "5c35a6f7-ada6-4695-a9d8-6e3d8560790a") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.027166 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.027145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-config-volume\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.027209 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.027166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-tmp-dir\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.030669 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.030650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/991799e8-945e-4c15-aeaf-a664b5f10700-cert\") pod \"ingress-canary-9zgzq\" (UID: \"991799e8-945e-4c15-aeaf-a664b5f10700\") " pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:25.038539 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.038517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkzc\" (UniqueName: \"kubernetes.io/projected/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-kube-api-access-9pkzc\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.039077 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.039056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkj4\" (UniqueName: \"kubernetes.io/projected/991799e8-945e-4c15-aeaf-a664b5f10700-kube-api-access-bfkj4\") pod \"ingress-canary-9zgzq\" (UID: \"991799e8-945e-4c15-aeaf-a664b5f10700\") " pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:25.091417 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.091364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9zgzq"
Sep 19 08:21:25.228557 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.228523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:25.228683 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.228565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:25.228784 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.228681    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:25.228784 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.228723    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:25.228784 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.228761    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret podName:de943430-0a59-4ea7-b59e-93636d87c4a0 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:57.228744269 +0000 UTC m=+67.893879814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret") pod "global-pull-secret-syncer-bx8qb" (UID: "de943430-0a59-4ea7-b59e-93636d87c4a0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:25.228784 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.228776    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs podName:ff198034-e56f-44da-8427-9046ff196291 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:57.228769517 +0000 UTC m=+67.893905041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs") pod "network-metrics-daemon-tz7f6" (UID: "ff198034-e56f-44da-8427-9046ff196291") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:25.329759 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.329732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:25.329888 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.329873    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:25.329938 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.329894    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:25.329938 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.329902    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8czh7 for pod openshift-network-diagnostics/network-check-target-tqgz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:25.330002 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.329941    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7 podName:eadec7b8-90a1-465f-8731-eed5d4bdfd49 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:57.329930613 +0000 UTC m=+67.995066137 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8czh7" (UniqueName: "kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7") pod "network-check-target-tqgz5" (UID: "eadec7b8-90a1-465f-8731-eed5d4bdfd49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:25.376809 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.376730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9zgzq"]
Sep 19 08:21:25.380580 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:21:25.380559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991799e8_945e_4c15_aeaf_a664b5f10700.slice/crio-5bfeb7f92a29757966c2de28866e2295383253ac593931bd4df2bd607b843598 WatchSource:0}: Error finding container 5bfeb7f92a29757966c2de28866e2295383253ac593931bd4df2bd607b843598: Status 404 returned error can't find the container with id 5bfeb7f92a29757966c2de28866e2295383253ac593931bd4df2bd607b843598
Sep 19 08:21:25.530706 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.530678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:25.530841 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.530823    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.530897 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:25.530888    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls podName:5c35a6f7-ada6-4695-a9d8-6e3d8560790a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:26.53087424 +0000 UTC m=+37.196009765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls") pod "dns-default-84qvd" (UID: "5c35a6f7-ada6-4695-a9d8-6e3d8560790a") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.767554 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.767492    2579 generic.go:356] "Generic (PLEG): container finished" podID="193907a3-f925-4deb-a7c5-4707d8ee6788" containerID="5c7c85a3c96bd01aac2f8311acd9416d3ec9b1057cb4323a20012df0c864a605" exitCode=0
Sep 19 08:21:25.767934 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.767563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerDied","Data":"5c7c85a3c96bd01aac2f8311acd9416d3ec9b1057cb4323a20012df0c864a605"}
Sep 19 08:21:25.768774 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:25.768757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9zgzq" event={"ID":"991799e8-945e-4c15-aeaf-a664b5f10700","Type":"ContainerStarted","Data":"5bfeb7f92a29757966c2de28866e2295383253ac593931bd4df2bd607b843598"}
Sep 19 08:21:26.537205 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.537170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:26.537365 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:26.537327    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:26.537435 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:26.537389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls podName:5c35a6f7-ada6-4695-a9d8-6e3d8560790a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:28.537371681 +0000 UTC m=+39.202507215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls") pod "dns-default-84qvd" (UID: "5c35a6f7-ada6-4695-a9d8-6e3d8560790a") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:26.598589 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.598559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:26.598760 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.598660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:26.598760 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.598689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:26.601469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.601428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:26.601469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.601446    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:26.602387 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.602365    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c9bjw\""
Sep 19 08:21:26.602503 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.602376    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:26.602503 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.602403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47kpw\""
Sep 19 08:21:26.602503 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.602427    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:26.775639 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.775606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" event={"ID":"193907a3-f925-4deb-a7c5-4707d8ee6788","Type":"ContainerStarted","Data":"903195495ab9f566e64bb9730a09986900a2faadfa23979b77bb952d2847ec8e"}
Sep 19 08:21:26.801149 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:26.801100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x6bcm" podStartSLOduration=9.755077819 podStartE2EDuration="35.801082772s" totalImagesPullingTime="26.046004953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:26.799885603 +0000 UTC m=+37.465021148" watchObservedRunningTime="2026-09-19 08:21:26.801082772 +0000 UTC m=+37.466218352"
Sep 19 08:21:28.551527 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:28.551310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:28.552034 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:28.551452    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:28.552034 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:28.551613    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls podName:5c35a6f7-ada6-4695-a9d8-6e3d8560790a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:32.551597843 +0000 UTC m=+43.216733367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls") pod "dns-default-84qvd" (UID: "5c35a6f7-ada6-4695-a9d8-6e3d8560790a") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:28.781766 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:28.781733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9zgzq" event={"ID":"991799e8-945e-4c15-aeaf-a664b5f10700","Type":"ContainerStarted","Data":"dcc9d28c76afb26e2fcb07e8491cb6de33819418b133669335dedef49239e959"}
Sep 19 08:21:32.577931 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:32.577895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:32.578279 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:32.578021    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:32.578279 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:32.578098    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls podName:5c35a6f7-ada6-4695-a9d8-6e3d8560790a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.578085411 +0000 UTC m=+51.243220939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls") pod "dns-default-84qvd" (UID: "5c35a6f7-ada6-4695-a9d8-6e3d8560790a") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:38.149511 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.149407    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9zgzq" podStartSLOduration=11.770577413 podStartE2EDuration="14.149390708s" totalImagesPullingTime="2.378813295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:28.806370252 +0000 UTC m=+39.471505802" watchObservedRunningTime="2026-09-19 08:21:38.149390708 +0000 UTC m=+48.814526255"
Sep 19 08:21:38.150300 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.150283    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"]
Sep 19 08:21:38.190785 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.190760    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"]
Sep 19 08:21:38.190911 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.190848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.193116 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.193096    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 19 08:21:38.193214 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.193165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kz28n\""
Sep 19 08:21:38.193397 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.193383    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 19 08:21:38.214634 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.214614    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"]
Sep 19 08:21:38.214768 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.214753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"
Sep 19 08:21:38.216756 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.216732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tss9r\""
Sep 19 08:21:38.230774 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.230751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"]
Sep 19 08:21:38.230857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.230781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"]
Sep 19 08:21:38.230857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.230799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"]
Sep 19 08:21:38.230968 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.230889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.234125 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.234109    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 19 08:21:38.235120 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.235106    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 08:21:38.235333 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.235298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 08:21:38.235431 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.235340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cjf9t\""
Sep 19 08:21:38.235431 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.235351    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 19 08:21:38.272037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.269855    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"]
Sep 19 08:21:38.298726 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.298685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76cf86969b-jt5mj"]
Sep 19 08:21:38.298809 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.298786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:38.303222 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.303189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 19 08:21:38.303446 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.303431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gqmdp\""
Sep 19 08:21:38.303804 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.303790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 19 08:21:38.303870 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.303852    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:21:38.310266 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.310245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5eed00ec-765a-4ff5-ada9-02072cf59fb3-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.310341 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.310314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.310411 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.310390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bwr\" (UniqueName: \"kubernetes.io/projected/80301f23-031b-488e-991d-efc52d87ccc6-kube-api-access-85bwr\") pod \"network-check-source-7cbc45fddc-8mrzh\" (UID: \"80301f23-031b-488e-991d-efc52d87ccc6\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"
Sep 19 08:21:38.318611 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.318596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"]
Sep 19 08:21:38.318684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.318622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76cf86969b-jt5mj"]
Sep 19 08:21:38.318684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.318680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.322339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.322303    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 19 08:21:38.322459 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.322444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 19 08:21:38.322895 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.322882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-77sq5\""
Sep 19 08:21:38.322941 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.322898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 08:21:38.323257 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.323239    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 08:21:38.323552 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.323531    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 19 08:21:38.346367 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.346347    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 19 08:21:38.411403 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5eed00ec-765a-4ff5-ada9-02072cf59fb3-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.411403 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:38.411509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbfs4\" (UniqueName: \"kubernetes.io/projected/8a190dfe-d0b5-426c-a225-ad3caed916f1-kube-api-access-xbfs4\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.411509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.411509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bwr\" (UniqueName: \"kubernetes.io/projected/80301f23-031b-488e-991d-efc52d87ccc6-kube-api-access-85bwr\") pod \"network-check-source-7cbc45fddc-8mrzh\" (UID: \"80301f23-031b-488e-991d-efc52d87ccc6\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"
Sep 19 08:21:38.411509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-default-certificate\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.411625 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.411526    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:38.411625 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5ds\" (UniqueName: \"kubernetes.io/projected/36193d7a-8e53-4f3a-8214-876bb5d56eec-kube-api-access-gn5ds\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:38.411625 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.411594    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert podName:5eed00ec-765a-4ff5-ada9-02072cf59fb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:38.911574831 +0000 UTC m=+49.576710369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-c8ln2" (UID: "5eed00ec-765a-4ff5-ada9-02072cf59fb3") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:38.411738 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.411738 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmwg\" (UniqueName: \"kubernetes.io/projected/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-kube-api-access-wfmwg\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.411738 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.411830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.411830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-stats-auth\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.411830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.411992 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.411976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5eed00ec-765a-4ff5-ada9-02072cf59fb3-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.421625 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.421606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bwr\" (UniqueName: \"kubernetes.io/projected/80301f23-031b-488e-991d-efc52d87ccc6-kube-api-access-85bwr\") pod \"network-check-source-7cbc45fddc-8mrzh\" (UID: \"80301f23-031b-488e-991d-efc52d87ccc6\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"
Sep 19 08:21:38.512757 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbfs4\" (UniqueName: \"kubernetes.io/projected/8a190dfe-d0b5-426c-a225-ad3caed916f1-kube-api-access-xbfs4\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.512839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-default-certificate\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.512839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5ds\" (UniqueName: \"kubernetes.io/projected/36193d7a-8e53-4f3a-8214-876bb5d56eec-kube-api-access-gn5ds\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:38.512918 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.512918 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmwg\" (UniqueName: \"kubernetes.io/projected/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-kube-api-access-wfmwg\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.512918 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.513051 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.513051 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-stats-auth\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.513051 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.512979    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:38.513051 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.512989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.513051 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.513042    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls podName:07e0274e-0b96-4df0-a08e-6c1fe4ea9172 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.013023746 +0000 UTC m=+49.678159287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-x7s9q" (UID: "07e0274e-0b96-4df0-a08e-6c1fe4ea9172") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:38.513281 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.513065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:38.513281 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.513092    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.013072386 +0000 UTC m=+49.678207931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:38.513281 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.513096    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:38.513281 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.513140    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.013129119 +0000 UTC m=+49.678264643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : secret "router-metrics-certs-default" not found
Sep 19 08:21:38.513281 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.513155    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:38.513281 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.513194    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls podName:36193d7a-8e53-4f3a-8214-876bb5d56eec nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.013181676 +0000 UTC m=+49.678317221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-85c5b" (UID: "36193d7a-8e53-4f3a-8214-876bb5d56eec") : secret "samples-operator-tls" not found
Sep 19 08:21:38.513650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.513614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.515067 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.515051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-default-certificate\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.515254 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.515236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-stats-auth\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.522071 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.522046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"
Sep 19 08:21:38.523855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.523819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmwg\" (UniqueName: \"kubernetes.io/projected/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-kube-api-access-wfmwg\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:38.525045 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.525026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbfs4\" (UniqueName: \"kubernetes.io/projected/8a190dfe-d0b5-426c-a225-ad3caed916f1-kube-api-access-xbfs4\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:38.525539 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.525519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5ds\" (UniqueName: \"kubernetes.io/projected/36193d7a-8e53-4f3a-8214-876bb5d56eec-kube-api-access-gn5ds\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:38.815541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.815513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh"]
Sep 19 08:21:38.818334 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:21:38.818312    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80301f23_031b_488e_991d_efc52d87ccc6.slice/crio-6e1c6f37a0d5cd024e52cbd29ef00d318d38900985e4fbc05959327285a1cb96 WatchSource:0}: Error finding container 6e1c6f37a0d5cd024e52cbd29ef00d318d38900985e4fbc05959327285a1cb96: Status 404 returned error can't find the container with id 6e1c6f37a0d5cd024e52cbd29ef00d318d38900985e4fbc05959327285a1cb96
Sep 19 08:21:38.914983 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:38.914962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:38.915119 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.915101    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:38.915180 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:38.915171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert podName:5eed00ec-765a-4ff5-ada9-02072cf59fb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.915156843 +0000 UTC m=+50.580292367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-c8ln2" (UID: "5eed00ec-765a-4ff5-ada9-02072cf59fb3") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:39.015753 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:39.015731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:39.015853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:39.015765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:39.015853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:39.015821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:39.015918 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.015899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.015879366 +0000 UTC m=+50.681014897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:39.016014 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.015946    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:39.016014 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:39.015972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:39.016014 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.015998    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.015984434 +0000 UTC m=+50.681119957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : secret "router-metrics-certs-default" not found
Sep 19 08:21:39.016014 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.015951    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:39.016145 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.016035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls podName:36193d7a-8e53-4f3a-8214-876bb5d56eec nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.016025941 +0000 UTC m=+50.681161466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-85c5b" (UID: "36193d7a-8e53-4f3a-8214-876bb5d56eec") : secret "samples-operator-tls" not found
Sep 19 08:21:39.016145 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.016037    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:39.016145 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.016067    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls podName:07e0274e-0b96-4df0-a08e-6c1fe4ea9172 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.016058901 +0000 UTC m=+50.681194430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-x7s9q" (UID: "07e0274e-0b96-4df0-a08e-6c1fe4ea9172") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:39.809471 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:39.809439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh" event={"ID":"80301f23-031b-488e-991d-efc52d87ccc6","Type":"ContainerStarted","Data":"6e1c6f37a0d5cd024e52cbd29ef00d318d38900985e4fbc05959327285a1cb96"}
Sep 19 08:21:39.924167 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:39.924133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:39.924319 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.924274    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:39.924360 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:39.924339    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert podName:5eed00ec-765a-4ff5-ada9-02072cf59fb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:41.924322167 +0000 UTC m=+52.589457693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-c8ln2" (UID: "5eed00ec-765a-4ff5-ada9-02072cf59fb3") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:40.024910 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:40.024876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:40.025065 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:40.024928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:40.025065 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025027    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:40.025065 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025055    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.025036272 +0000 UTC m=+52.690171801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:40.025233 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.025073082 +0000 UTC m=+52.690208609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : secret "router-metrics-certs-default" not found
Sep 19 08:21:40.025233 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:40.025050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:40.025233 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025129    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:40.025233 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls podName:36193d7a-8e53-4f3a-8214-876bb5d56eec nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.025168315 +0000 UTC m=+52.690303860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-85c5b" (UID: "36193d7a-8e53-4f3a-8214-876bb5d56eec") : secret "samples-operator-tls" not found
Sep 19 08:21:40.025419 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:40.025241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:40.025419 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025313    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:40.025419 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.025364    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls podName:07e0274e-0b96-4df0-a08e-6c1fe4ea9172 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.025351756 +0000 UTC m=+52.690487297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-x7s9q" (UID: "07e0274e-0b96-4df0-a08e-6c1fe4ea9172") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:40.629322 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:40.629282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:40.629498 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.629407    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:40.629498 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:40.629471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls podName:5c35a6f7-ada6-4695-a9d8-6e3d8560790a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:56.629449547 +0000 UTC m=+67.294585094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls") pod "dns-default-84qvd" (UID: "5c35a6f7-ada6-4695-a9d8-6e3d8560790a") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:41.941962 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:41.941900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:41.942227 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:41.942032    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:41.942227 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:41.942084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert podName:5eed00ec-765a-4ff5-ada9-02072cf59fb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:45.942068436 +0000 UTC m=+56.607203967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-c8ln2" (UID: "5eed00ec-765a-4ff5-ada9-02072cf59fb3") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:42.042748 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.042710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:42.042748 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.042752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.042775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.042799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042842    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042856    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.042872843 +0000 UTC m=+56.708008370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042915    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042923    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls podName:07e0274e-0b96-4df0-a08e-6c1fe4ea9172 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.042912899 +0000 UTC m=+56.708048423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-x7s9q" (UID: "07e0274e-0b96-4df0-a08e-6c1fe4ea9172") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042943    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.042932742 +0000 UTC m=+56.708068267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : secret "router-metrics-certs-default" not found
Sep 19 08:21:42.042955 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:42.042956    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls podName:36193d7a-8e53-4f3a-8214-876bb5d56eec nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.042949365 +0000 UTC m=+56.708084890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-85c5b" (UID: "36193d7a-8e53-4f3a-8214-876bb5d56eec") : secret "samples-operator-tls" not found
Sep 19 08:21:42.818197 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.818159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh" event={"ID":"80301f23-031b-488e-991d-efc52d87ccc6","Type":"ContainerStarted","Data":"2b6af79c6c282061b71562e90b3cdf22e8c3790ad7664416b34735cf91a2250d"}
Sep 19 08:21:42.847009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.846964    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8mrzh" podStartSLOduration=1.8273430560000001 podStartE2EDuration="4.846950695s" totalImagesPullingTime="3.019607639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:42.846562142 +0000 UTC m=+53.511697689" watchObservedRunningTime="2026-09-19 08:21:42.846950695 +0000 UTC m=+53.512086295"
Sep 19 08:21:42.906198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.906167    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jvd6q"]
Sep 19 08:21:42.927235 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.927212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvd6q"]
Sep 19 08:21:42.927369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.927325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:42.929741 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.929707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 08:21:42.929867 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.929831    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 08:21:42.929935 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.929900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 08:21:42.929935 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.929926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 08:21:42.929935 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:42.929931    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k69r7\""
Sep 19 08:21:43.049441 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.049420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596bc\" (UniqueName: \"kubernetes.io/projected/e58663f3-4077-4718-84d0-016a4127b3f5-kube-api-access-596bc\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.049835 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.049449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.049835 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.049536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58663f3-4077-4718-84d0-016a4127b3f5-data-volume\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.049835 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.049609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.049835 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.049649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58663f3-4077-4718-84d0-016a4127b3f5-crio-socket\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.150545 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58663f3-4077-4718-84d0-016a4127b3f5-data-volume\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.150651 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.150651 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58663f3-4077-4718-84d0-016a4127b3f5-crio-socket\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.150651 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596bc\" (UniqueName: \"kubernetes.io/projected/e58663f3-4077-4718-84d0-016a4127b3f5-kube-api-access-596bc\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.150839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.150839 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.150672    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.150839 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.150746    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:43.650726141 +0000 UTC m=+54.315861668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.150839 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.150790    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.150839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58663f3-4077-4718-84d0-016a4127b3f5-crio-socket\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.151081 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.150851    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:43.65083296 +0000 UTC m=+54.315968485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.151081 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.150863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58663f3-4077-4718-84d0-016a4127b3f5-data-volume\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.158316 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.158299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-596bc\" (UniqueName: \"kubernetes.io/projected/e58663f3-4077-4718-84d0-016a4127b3f5-kube-api-access-596bc\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.655205 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.655172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.655366 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:43.655224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:43.655366 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.655293    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.655366 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.655314    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.655366 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.655352    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:44.655337893 +0000 UTC m=+55.320473416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.655366 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:43.655368    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:44.655362055 +0000 UTC m=+55.320497578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:44.662224 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:44.662189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:44.662592 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:44.662250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:44.662592 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:44.662316    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:44.662592 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:44.662385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.662366917 +0000 UTC m=+57.327502459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:44.662592 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:44.662389    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:44.662592 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:44.662427    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.662416499 +0000 UTC m=+57.327552037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:44.831976 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:44.831950    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wwjml_add4b376-afa3-493c-aeb1-916bfeacbf89/dns-node-resolver/0.log"
Sep 19 08:21:45.970259 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:45.970226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:45.970726 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:45.970394    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:45.970726 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:45.970468    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert podName:5eed00ec-765a-4ff5-ada9-02072cf59fb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:53.970449187 +0000 UTC m=+64.635584714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-c8ln2" (UID: "5eed00ec-765a-4ff5-ada9-02072cf59fb3") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:46.032295 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.032275    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vmpk4_0a6f6ec9-e47c-4957-b28b-14014d1af467/node-ca/0.log"
Sep 19 08:21:46.071533 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.071508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:46.071626 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.071547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:46.071626 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.071583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:46.071626 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.071618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071644    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071668    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.0716501 +0000 UTC m=+64.736785624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071710    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls podName:07e0274e-0b96-4df0-a08e-6c1fe4ea9172 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.071681397 +0000 UTC m=+64.736816922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-x7s9q" (UID: "07e0274e-0b96-4df0-a08e-6c1fe4ea9172") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071740    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071750    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls podName:36193d7a-8e53-4f3a-8214-876bb5d56eec nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.071776953 +0000 UTC m=+64.736912480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-85c5b" (UID: "36193d7a-8e53-4f3a-8214-876bb5d56eec") : secret "samples-operator-tls" not found
Sep 19 08:21:46.071818 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.071809    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.071800492 +0000 UTC m=+64.736936022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : secret "router-metrics-certs-default" not found
Sep 19 08:21:46.632606 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.632575    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9zgzq_991799e8-945e-4c15-aeaf-a664b5f10700/serve-healthcheck-canary/0.log"
Sep 19 08:21:46.676840 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.676818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:46.676949 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:46.676923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:46.676985 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.676950    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:46.677027 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.676998    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:46.677027 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.677010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:50.676996408 +0000 UTC m=+61.342131936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:46.677109 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:46.677035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:50.677024993 +0000 UTC m=+61.342160521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:48.746772 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:48.746746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bkpds"
Sep 19 08:21:50.705587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:50.705555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:50.705967 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:50.705610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:50.705967 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:50.705678    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:50.705967 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:50.705754    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:50.705967 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:50.705757    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:58.705740405 +0000 UTC m=+69.370875949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:50.705967 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:50.705801    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:58.705790826 +0000 UTC m=+69.370926349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:54.029187 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.029141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:21:54.029550 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:54.029294    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:54.029550 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:54.029389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert podName:5eed00ec-765a-4ff5-ada9-02072cf59fb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:10.029373598 +0000 UTC m=+80.694509121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-c8ln2" (UID: "5eed00ec-765a-4ff5-ada9-02072cf59fb3") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:54.130228 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.130204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:21:54.130304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.130235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:54.130304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.130258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:54.130304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.130279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:54.130444 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:54.130337    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:54.130444 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:54.130390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls podName:07e0274e-0b96-4df0-a08e-6c1fe4ea9172 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:10.130376952 +0000 UTC m=+80.795512476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-x7s9q" (UID: "07e0274e-0b96-4df0-a08e-6c1fe4ea9172") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:54.130551 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:54.130532    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle podName:8a190dfe-d0b5-426c-a225-ad3caed916f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:10.130517038 +0000 UTC m=+80.795652587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle") pod "router-default-76cf86969b-jt5mj" (UID: "8a190dfe-d0b5-426c-a225-ad3caed916f1") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:54.132515 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.132492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a190dfe-d0b5-426c-a225-ad3caed916f1-metrics-certs\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:21:54.132644 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.132628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36193d7a-8e53-4f3a-8214-876bb5d56eec-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-85c5b\" (UID: \"36193d7a-8e53-4f3a-8214-876bb5d56eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:54.208824 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.208796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gqmdp\""
Sep 19 08:21:54.216971 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.216953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"
Sep 19 08:21:54.482486 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.482457    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b"]
Sep 19 08:21:54.844775 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:54.844737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b" event={"ID":"36193d7a-8e53-4f3a-8214-876bb5d56eec","Type":"ContainerStarted","Data":"a13ba6fc608151bb99d254a0a9d68a5208315cc802987fa4ea1e7ba9c7d41322"}
Sep 19 08:21:56.649100 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.649073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:56.651055 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.651039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c35a6f7-ada6-4695-a9d8-6e3d8560790a-metrics-tls\") pod \"dns-default-84qvd\" (UID: \"5c35a6f7-ada6-4695-a9d8-6e3d8560790a\") " pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:56.850801 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.850767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b" event={"ID":"36193d7a-8e53-4f3a-8214-876bb5d56eec","Type":"ContainerStarted","Data":"0d79f3e75e4ca24a169bba0e4276840c264e7798fe71609634018e8ee53ccb5d"}
Sep 19 08:21:56.850966 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.850842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b" event={"ID":"36193d7a-8e53-4f3a-8214-876bb5d56eec","Type":"ContainerStarted","Data":"993ba16552b3067de896b3ceb0c346f758a8c6d9bacff9c2e949640e625a0f1b"}
Sep 19 08:21:56.867441 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.867395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-85c5b" podStartSLOduration=16.920088055 podStartE2EDuration="18.86738193s" totalImagesPullingTime="1.947293875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:56.866742834 +0000 UTC m=+67.531878382" watchObservedRunningTime="2026-09-19 08:21:56.86738193 +0000 UTC m=+67.532517476"
Sep 19 08:21:56.882010 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.881995    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g2bvg\""
Sep 19 08:21:56.890814 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:56.890796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:57.176784 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.176711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-84qvd"]
Sep 19 08:21:57.179934 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:21:57.179911    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c35a6f7_ada6_4695_a9d8_6e3d8560790a.slice/crio-7c93548809d4e2e56101c18b1c0ff58439e2f908a6a102be7c163f7bfb96bc41 WatchSource:0}: Error finding container 7c93548809d4e2e56101c18b1c0ff58439e2f908a6a102be7c163f7bfb96bc41: Status 404 returned error can't find the container with id 7c93548809d4e2e56101c18b1c0ff58439e2f908a6a102be7c163f7bfb96bc41
Sep 19 08:21:57.253600 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.253575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:57.253733 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.253626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:57.256177 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.256156    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:57.256258 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.256159    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:57.266146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.266121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff198034-e56f-44da-8427-9046ff196291-metrics-certs\") pod \"network-metrics-daemon-tz7f6\" (UID: \"ff198034-e56f-44da-8427-9046ff196291\") " pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:57.266146 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.266140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de943430-0a59-4ea7-b59e-93636d87c4a0-original-pull-secret\") pod \"global-pull-secret-syncer-bx8qb\" (UID: \"de943430-0a59-4ea7-b59e-93636d87c4a0\") " pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:57.354663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.354642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:57.356713 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.356674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czh7\" (UniqueName: \"kubernetes.io/projected/eadec7b8-90a1-465f-8731-eed5d4bdfd49-kube-api-access-8czh7\") pod \"network-check-target-tqgz5\" (UID: \"eadec7b8-90a1-465f-8731-eed5d4bdfd49\") " pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:57.511062 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.511005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx8qb"
Sep 19 08:21:57.519365 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.519348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47kpw\""
Sep 19 08:21:57.525123 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.525107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c9bjw\""
Sep 19 08:21:57.527227 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.527215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:57.533625 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.533609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tz7f6"
Sep 19 08:21:57.797430 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.797389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx8qb"]
Sep 19 08:21:57.800584 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:21:57.800559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde943430_0a59_4ea7_b59e_93636d87c4a0.slice/crio-714e5d0670411b8fe83d3f3d2fcf6d332d92f5ca9b388aed9a759d434e252afb WatchSource:0}: Error finding container 714e5d0670411b8fe83d3f3d2fcf6d332d92f5ca9b388aed9a759d434e252afb: Status 404 returned error can't find the container with id 714e5d0670411b8fe83d3f3d2fcf6d332d92f5ca9b388aed9a759d434e252afb
Sep 19 08:21:57.820057 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.820031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqgz5"]
Sep 19 08:21:57.822947 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:21:57.822921    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadec7b8_90a1_465f_8731_eed5d4bdfd49.slice/crio-87285dad2af148368a3725a2e037460126225db12c4f67d6e5026ab9cfe310e9 WatchSource:0}: Error finding container 87285dad2af148368a3725a2e037460126225db12c4f67d6e5026ab9cfe310e9: Status 404 returned error can't find the container with id 87285dad2af148368a3725a2e037460126225db12c4f67d6e5026ab9cfe310e9
Sep 19 08:21:57.845808 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.845749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tz7f6"]
Sep 19 08:21:57.848063 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:21:57.848043    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff198034_e56f_44da_8427_9046ff196291.slice/crio-c35e3a60c07008c5f4872340b433a02090a87771f22455f73204cae3041bf004 WatchSource:0}: Error finding container c35e3a60c07008c5f4872340b433a02090a87771f22455f73204cae3041bf004: Status 404 returned error can't find the container with id c35e3a60c07008c5f4872340b433a02090a87771f22455f73204cae3041bf004
Sep 19 08:21:57.854853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.854828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tz7f6" event={"ID":"ff198034-e56f-44da-8427-9046ff196291","Type":"ContainerStarted","Data":"c35e3a60c07008c5f4872340b433a02090a87771f22455f73204cae3041bf004"}
Sep 19 08:21:57.856077 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.856051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-84qvd" event={"ID":"5c35a6f7-ada6-4695-a9d8-6e3d8560790a","Type":"ContainerStarted","Data":"7c93548809d4e2e56101c18b1c0ff58439e2f908a6a102be7c163f7bfb96bc41"}
Sep 19 08:21:57.857386 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.857356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqgz5" event={"ID":"eadec7b8-90a1-465f-8731-eed5d4bdfd49","Type":"ContainerStarted","Data":"87285dad2af148368a3725a2e037460126225db12c4f67d6e5026ab9cfe310e9"}
Sep 19 08:21:57.858528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:57.858504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx8qb" event={"ID":"de943430-0a59-4ea7-b59e-93636d87c4a0","Type":"ContainerStarted","Data":"714e5d0670411b8fe83d3f3d2fcf6d332d92f5ca9b388aed9a759d434e252afb"}
Sep 19 08:21:58.766589 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:58.766551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:58.766800 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:58.766671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:58.766800 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:58.766724    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:58.766800 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:21:58.766801    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls podName:e58663f3-4077-4718-84d0-016a4127b3f5 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:14.766779344 +0000 UTC m=+85.431914875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvd6q" (UID: "e58663f3-4077-4718-84d0-016a4127b3f5") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:58.767284 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:58.767261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58663f3-4077-4718-84d0-016a4127b3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:21:58.864269 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:58.864224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqgz5" event={"ID":"eadec7b8-90a1-465f-8731-eed5d4bdfd49","Type":"ContainerStarted","Data":"a71d50557d8619c93a24983a64e3715a87795c1235df8f6b3b4ea6ad4498a00d"}
Sep 19 08:21:58.864741 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:58.864373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:21:58.881096 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:58.881053    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tqgz5" podStartSLOduration=67.88103903 podStartE2EDuration="1m7.88103903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:58.880656867 +0000 UTC m=+69.545792414" watchObservedRunningTime="2026-09-19 08:21:58.88103903 +0000 UTC m=+69.546174577"
Sep 19 08:21:59.869523 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.869481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tz7f6" event={"ID":"ff198034-e56f-44da-8427-9046ff196291","Type":"ContainerStarted","Data":"935928a2a2ebc62e1b25789f8b4b3c7cf7e9b6678d9fa900164db35baded3a06"}
Sep 19 08:21:59.869981 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.869554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tz7f6" event={"ID":"ff198034-e56f-44da-8427-9046ff196291","Type":"ContainerStarted","Data":"c2675d3d419821297ca42053c0d5fa7529d790bd6ca00970ebed8309e9de2477"}
Sep 19 08:21:59.871521 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.871492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-84qvd" event={"ID":"5c35a6f7-ada6-4695-a9d8-6e3d8560790a","Type":"ContainerStarted","Data":"06177bb21c9119d58aeff70834adcfc5a365b69ef2c0e3bdc6baf6bf18270cac"}
Sep 19 08:21:59.871662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.871540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-84qvd" event={"ID":"5c35a6f7-ada6-4695-a9d8-6e3d8560790a","Type":"ContainerStarted","Data":"037b746b1c20f3caa403a8cd293c50b340c43d957a77c3a27e063a97eb4c68a2"}
Sep 19 08:21:59.871746 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.871678    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-84qvd"
Sep 19 08:21:59.886419 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.886372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tz7f6" podStartSLOduration=67.563350504 podStartE2EDuration="1m8.886355223s" totalImagesPullingTime="1.323004719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:59.885660223 +0000 UTC m=+70.550795769" watchObservedRunningTime="2026-09-19 08:21:59.886355223 +0000 UTC m=+70.551490763"
Sep 19 08:21:59.901179 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:21:59.901134    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-84qvd" podStartSLOduration=33.935320283 podStartE2EDuration="35.901117204s" totalImagesPullingTime="1.965796921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:59.900771658 +0000 UTC m=+70.565907217" watchObservedRunningTime="2026-09-19 08:21:59.901117204 +0000 UTC m=+70.566252763"
Sep 19 08:22:01.886904 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:01.886861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx8qb" event={"ID":"de943430-0a59-4ea7-b59e-93636d87c4a0","Type":"ContainerStarted","Data":"d5eb669e7678ec493ffdddbcf5d83498dbcf843002c7027bbb7927e1850f3caf"}
Sep 19 08:22:05.668240 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.668104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bx8qb" podStartSLOduration=70.75523721 podStartE2EDuration="1m14.668087355s" totalImagesPullingTime="3.912850145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:01.912734484 +0000 UTC m=+72.577870029" watchObservedRunningTime="2026-09-19 08:22:05.668087355 +0000 UTC m=+76.333222879"
Sep 19 08:22:05.669183 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.669165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"]
Sep 19 08:22:05.671052 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.671038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.674318 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.674296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hxs7g\""
Sep 19 08:22:05.674424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.674339    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 08:22:05.674424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.674338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 08:22:05.674424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.674338    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 19 08:22:05.674544 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.674296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 08:22:05.693039 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.693018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"]
Sep 19 08:22:05.721940 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.721914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86e47ae1-783d-4485-9465-d1e43b19edd7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-779b47cc96-8kdk6\" (UID: \"86e47ae1-783d-4485-9465-d1e43b19edd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.722069 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.721953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7x9\" (UniqueName: \"kubernetes.io/projected/86e47ae1-783d-4485-9465-d1e43b19edd7-kube-api-access-2h7x9\") pod \"managed-serviceaccount-addon-agent-779b47cc96-8kdk6\" (UID: \"86e47ae1-783d-4485-9465-d1e43b19edd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.823122 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.823097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86e47ae1-783d-4485-9465-d1e43b19edd7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-779b47cc96-8kdk6\" (UID: \"86e47ae1-783d-4485-9465-d1e43b19edd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.823270 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.823134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7x9\" (UniqueName: \"kubernetes.io/projected/86e47ae1-783d-4485-9465-d1e43b19edd7-kube-api-access-2h7x9\") pod \"managed-serviceaccount-addon-agent-779b47cc96-8kdk6\" (UID: \"86e47ae1-783d-4485-9465-d1e43b19edd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.825609 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.825578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86e47ae1-783d-4485-9465-d1e43b19edd7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-779b47cc96-8kdk6\" (UID: \"86e47ae1-783d-4485-9465-d1e43b19edd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.849730 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.849689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7x9\" (UniqueName: \"kubernetes.io/projected/86e47ae1-783d-4485-9465-d1e43b19edd7-kube-api-access-2h7x9\") pod \"managed-serviceaccount-addon-agent-779b47cc96-8kdk6\" (UID: \"86e47ae1-783d-4485-9465-d1e43b19edd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:05.989140 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:05.989074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"
Sep 19 08:22:06.283320 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:06.283294    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e47ae1_783d_4485_9465_d1e43b19edd7.slice/crio-c20b0ca9e50bda522298f0f384b54affce7ff8c77af1a16f9e1877130a5df327 WatchSource:0}: Error finding container c20b0ca9e50bda522298f0f384b54affce7ff8c77af1a16f9e1877130a5df327: Status 404 returned error can't find the container with id c20b0ca9e50bda522298f0f384b54affce7ff8c77af1a16f9e1877130a5df327
Sep 19 08:22:06.286529 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:06.286508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6"]
Sep 19 08:22:06.905652 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:06.905617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6" event={"ID":"86e47ae1-783d-4485-9465-d1e43b19edd7","Type":"ContainerStarted","Data":"c20b0ca9e50bda522298f0f384b54affce7ff8c77af1a16f9e1877130a5df327"}
Sep 19 08:22:08.913703 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:08.913654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6" event={"ID":"86e47ae1-783d-4485-9465-d1e43b19edd7","Type":"ContainerStarted","Data":"84b1996a642b52fa5c787cebdb7d17d425c6d16b84346e7e86773bbe94c34a02"}
Sep 19 08:22:08.931430 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:08.931385    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-779b47cc96-8kdk6" podStartSLOduration=1.8026227430000001 podStartE2EDuration="3.931371846s" totalImagesPullingTime="2.128749103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:08.930607203 +0000 UTC m=+79.595742750" watchObservedRunningTime="2026-09-19 08:22:08.931371846 +0000 UTC m=+79.596507392"
Sep 19 08:22:09.881627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:09.881600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-84qvd"
Sep 19 08:22:10.052825 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.052783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:22:10.054965 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.054946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5eed00ec-765a-4ff5-ada9-02072cf59fb3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-c8ln2\" (UID: \"5eed00ec-765a-4ff5-ada9-02072cf59fb3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:22:10.153758 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.153706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:22:10.153758 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.153734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:10.154216 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.154197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a190dfe-d0b5-426c-a225-ad3caed916f1-service-ca-bundle\") pod \"router-default-76cf86969b-jt5mj\" (UID: \"8a190dfe-d0b5-426c-a225-ad3caed916f1\") " pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:10.155764 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.155746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/07e0274e-0b96-4df0-a08e-6c1fe4ea9172-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-x7s9q\" (UID: \"07e0274e-0b96-4df0-a08e-6c1fe4ea9172\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:22:10.302328 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.302303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kz28n\""
Sep 19 08:22:10.310301 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.310283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"
Sep 19 08:22:10.340795 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.340768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cjf9t\""
Sep 19 08:22:10.348736 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.348713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"
Sep 19 08:22:10.428975 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.428912    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-77sq5\""
Sep 19 08:22:10.437062 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.437047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:10.573824 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.573796    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2"]
Sep 19 08:22:10.576337 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:10.576312    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eed00ec_765a_4ff5_ada9_02072cf59fb3.slice/crio-f29b3f9b39de5d160faa8abfec6741b202e4f1886f84157566c0a27b19e9dcd1 WatchSource:0}: Error finding container f29b3f9b39de5d160faa8abfec6741b202e4f1886f84157566c0a27b19e9dcd1: Status 404 returned error can't find the container with id f29b3f9b39de5d160faa8abfec6741b202e4f1886f84157566c0a27b19e9dcd1
Sep 19 08:22:10.617909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.617832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q"]
Sep 19 08:22:10.621131 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:10.621107    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e0274e_0b96_4df0_a08e_6c1fe4ea9172.slice/crio-58f9718805fd5233e8a84aedf32df160ed3867c79e1546a90992fb33b5033ec5 WatchSource:0}: Error finding container 58f9718805fd5233e8a84aedf32df160ed3867c79e1546a90992fb33b5033ec5: Status 404 returned error can't find the container with id 58f9718805fd5233e8a84aedf32df160ed3867c79e1546a90992fb33b5033ec5
Sep 19 08:22:10.701903 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.701841    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76cf86969b-jt5mj"]
Sep 19 08:22:10.705492 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:10.705469    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a190dfe_d0b5_426c_a225_ad3caed916f1.slice/crio-47550ad984d3e42dc620ddb9a2cf93cfad7cd61b92803919ff678c8ca81d2135 WatchSource:0}: Error finding container 47550ad984d3e42dc620ddb9a2cf93cfad7cd61b92803919ff678c8ca81d2135: Status 404 returned error can't find the container with id 47550ad984d3e42dc620ddb9a2cf93cfad7cd61b92803919ff678c8ca81d2135
Sep 19 08:22:10.922246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.922212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76cf86969b-jt5mj" event={"ID":"8a190dfe-d0b5-426c-a225-ad3caed916f1","Type":"ContainerStarted","Data":"066289ee5acd21709cb154962a5d9b471e2ca42098092674c1e706b5fd8be90c"}
Sep 19 08:22:10.922440 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.922270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76cf86969b-jt5mj" event={"ID":"8a190dfe-d0b5-426c-a225-ad3caed916f1","Type":"ContainerStarted","Data":"47550ad984d3e42dc620ddb9a2cf93cfad7cd61b92803919ff678c8ca81d2135"}
Sep 19 08:22:10.923507 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.923481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q" event={"ID":"07e0274e-0b96-4df0-a08e-6c1fe4ea9172","Type":"ContainerStarted","Data":"58f9718805fd5233e8a84aedf32df160ed3867c79e1546a90992fb33b5033ec5"}
Sep 19 08:22:10.924673 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.924649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2" event={"ID":"5eed00ec-765a-4ff5-ada9-02072cf59fb3","Type":"ContainerStarted","Data":"f29b3f9b39de5d160faa8abfec6741b202e4f1886f84157566c0a27b19e9dcd1"}
Sep 19 08:22:10.944165 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:10.944121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76cf86969b-jt5mj" podStartSLOduration=32.944106947 podStartE2EDuration="32.944106947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:10.942462293 +0000 UTC m=+81.607597839" watchObservedRunningTime="2026-09-19 08:22:10.944106947 +0000 UTC m=+81.609242492"
Sep 19 08:22:11.438169 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:11.438134    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:11.441780 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:11.441755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:11.930380 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:11.930335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2" event={"ID":"5eed00ec-765a-4ff5-ada9-02072cf59fb3","Type":"ContainerStarted","Data":"63daa78b4886581f704f821834662e403413f7fc61322d730a99b63c01462927"}
Sep 19 08:22:11.930589 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:11.930569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:11.931984 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:11.931960    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76cf86969b-jt5mj"
Sep 19 08:22:11.952413 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:11.952365    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-c8ln2" podStartSLOduration=32.977304245 podStartE2EDuration="33.95235138s" totalImagesPullingTime="975.047135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:11.951507366 +0000 UTC m=+82.616642913" watchObservedRunningTime="2026-09-19 08:22:11.95235138 +0000 UTC m=+82.617486925"
Sep 19 08:22:12.817631 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.817603    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"]
Sep 19 08:22:12.819546 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.819528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:12.823133 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.823110    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gwtcm\""
Sep 19 08:22:12.823856 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.823832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 19 08:22:12.850029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.849997    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"]
Sep 19 08:22:12.874844 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.874820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/33c79094-d1a0-4861-a718-5cbfbb580b66-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7xvfv\" (UID: \"33c79094-d1a0-4861-a718-5cbfbb580b66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:12.935210 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.935182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q" event={"ID":"07e0274e-0b96-4df0-a08e-6c1fe4ea9172","Type":"ContainerStarted","Data":"c23aa04bce7b52c5e9c2c04fab290764f9340e01c198ed5eb22f5d698d8edb92"}
Sep 19 08:22:12.956536 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.956490    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-x7s9q" podStartSLOduration=33.2550113 podStartE2EDuration="34.956466416s" totalImagesPullingTime="1.701455116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:12.955482742 +0000 UTC m=+83.620618289" watchObservedRunningTime="2026-09-19 08:22:12.956466416 +0000 UTC m=+83.621601963"
Sep 19 08:22:12.975945 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:12.975924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/33c79094-d1a0-4861-a718-5cbfbb580b66-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7xvfv\" (UID: \"33c79094-d1a0-4861-a718-5cbfbb580b66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:12.976076 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:22:12.976055    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 19 08:22:12.976140 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:22:12.976125    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c79094-d1a0-4861-a718-5cbfbb580b66-tls-certificates podName:33c79094-d1a0-4861-a718-5cbfbb580b66 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:13.476107598 +0000 UTC m=+84.141243125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/33c79094-d1a0-4861-a718-5cbfbb580b66-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-7xvfv" (UID: "33c79094-d1a0-4861-a718-5cbfbb580b66") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 19 08:22:13.479801 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:13.479770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/33c79094-d1a0-4861-a718-5cbfbb580b66-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7xvfv\" (UID: \"33c79094-d1a0-4861-a718-5cbfbb580b66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:13.482063 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:13.482035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/33c79094-d1a0-4861-a718-5cbfbb580b66-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7xvfv\" (UID: \"33c79094-d1a0-4861-a718-5cbfbb580b66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:13.727955 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:13.727910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:13.994848 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:13.994743    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"]
Sep 19 08:22:13.997224 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:13.997201    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c79094_d1a0_4861_a718_5cbfbb580b66.slice/crio-5bb05fe62f749766bc668f68ae1ffdaa5ebed1cdffc85bb23393bceaf9acac78 WatchSource:0}: Error finding container 5bb05fe62f749766bc668f68ae1ffdaa5ebed1cdffc85bb23393bceaf9acac78: Status 404 returned error can't find the container with id 5bb05fe62f749766bc668f68ae1ffdaa5ebed1cdffc85bb23393bceaf9acac78
Sep 19 08:22:14.789925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:14.789891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:22:14.792470 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:14.792440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58663f3-4077-4718-84d0-016a4127b3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvd6q\" (UID: \"e58663f3-4077-4718-84d0-016a4127b3f5\") " pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:22:14.944728 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:14.944684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv" event={"ID":"33c79094-d1a0-4861-a718-5cbfbb580b66","Type":"ContainerStarted","Data":"5bb05fe62f749766bc668f68ae1ffdaa5ebed1cdffc85bb23393bceaf9acac78"}
Sep 19 08:22:15.038871 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.038843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k69r7\""
Sep 19 08:22:15.046828 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.046810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvd6q"
Sep 19 08:22:15.373734 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.373638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvd6q"]
Sep 19 08:22:15.377243 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:15.377209    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58663f3_4077_4718_84d0_016a4127b3f5.slice/crio-cbc5823596e63faa435e8b27c85a20f2aa757ded2b3d16158f31e292e4a1cae7 WatchSource:0}: Error finding container cbc5823596e63faa435e8b27c85a20f2aa757ded2b3d16158f31e292e4a1cae7: Status 404 returned error can't find the container with id cbc5823596e63faa435e8b27c85a20f2aa757ded2b3d16158f31e292e4a1cae7
Sep 19 08:22:15.949886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.949846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvd6q" event={"ID":"e58663f3-4077-4718-84d0-016a4127b3f5","Type":"ContainerStarted","Data":"697faea27a45f52d5b9431d5514fe624c9b00d4748209e2af04915259ef68000"}
Sep 19 08:22:15.950077 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.949910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvd6q" event={"ID":"e58663f3-4077-4718-84d0-016a4127b3f5","Type":"ContainerStarted","Data":"cbc5823596e63faa435e8b27c85a20f2aa757ded2b3d16158f31e292e4a1cae7"}
Sep 19 08:22:15.951261 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.951236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv" event={"ID":"33c79094-d1a0-4861-a718-5cbfbb580b66","Type":"ContainerStarted","Data":"e5b32977b56463396ec4387a4c4048260617939babf9afa205621b1c5fdae74a"}
Sep 19 08:22:15.951442 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.951417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:15.956203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.956183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv"
Sep 19 08:22:15.966510 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:15.966469    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7xvfv" podStartSLOduration=2.8546521780000003 podStartE2EDuration="3.966457791s" totalImagesPullingTime="1.111805613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:15.966270486 +0000 UTC m=+86.631406034" watchObservedRunningTime="2026-09-19 08:22:15.966457791 +0000 UTC m=+86.631593336"
Sep 19 08:22:16.560321 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.560291    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-845f7b946c-bgn6f"]
Sep 19 08:22:16.562277 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.562262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.564653 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.564632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 19 08:22:16.564767 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.564632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 19 08:22:16.565561 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.565542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 19 08:22:16.565634 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.565616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7kqxm\""
Sep 19 08:22:16.565874 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.565861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 19 08:22:16.565969 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.565951    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 19 08:22:16.566001 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.565966    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 19 08:22:16.566033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.566023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 19 08:22:16.572889 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.572869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845f7b946c-bgn6f"]
Sep 19 08:22:16.604436 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.604407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-oauth-config\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.604585 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.604451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-config\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.604585 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.604553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-oauth-serving-cert\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.604731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.604589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-service-ca\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.604731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.604632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76r9z\" (UniqueName: \"kubernetes.io/projected/89b9e6a4-027a-4b92-ae8b-a3df4835678f-kube-api-access-76r9z\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.604731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.604664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-serving-cert\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.705360 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.705331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-oauth-serving-cert\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.705360 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.705365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-service-ca\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.705533 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.705386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76r9z\" (UniqueName: \"kubernetes.io/projected/89b9e6a4-027a-4b92-ae8b-a3df4835678f-kube-api-access-76r9z\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.705533 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.705412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-serving-cert\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.705533 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.705439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-oauth-config\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.705533 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.705457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-config\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.706115 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.706090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-oauth-serving-cert\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.706246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.706094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-service-ca\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.706246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.706215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-config\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.707917 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.707890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-serving-cert\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.707917 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.707908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-oauth-config\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.713002 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.712976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76r9z\" (UniqueName: \"kubernetes.io/projected/89b9e6a4-027a-4b92-ae8b-a3df4835678f-kube-api-access-76r9z\") pod \"console-845f7b946c-bgn6f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") " pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.859285 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.859200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"]
Sep 19 08:22:16.861953 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.861934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:16.864562 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.864539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 19 08:22:16.864850 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.864823    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7jldd\""
Sep 19 08:22:16.865100 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.865082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 19 08:22:16.865188 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.865154    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 08:22:16.871331 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.871311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:16.883047 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.883025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"]
Sep 19 08:22:16.907313 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.907265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4bbbc45-66e4-41c2-bb88-859345dfd29c-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:16.907471 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.907323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4bbbc45-66e4-41c2-bb88-859345dfd29c-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:16.907471 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.907392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4bbbc45-66e4-41c2-bb88-859345dfd29c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:16.907586 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.907483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckjr\" (UniqueName: \"kubernetes.io/projected/f4bbbc45-66e4-41c2-bb88-859345dfd29c-kube-api-access-4ckjr\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:16.957330 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:16.957292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvd6q" event={"ID":"e58663f3-4077-4718-84d0-016a4127b3f5","Type":"ContainerStarted","Data":"02a6228937dbfd47d1a1df12aa4eb55b4d6543fcf7d6b31bdba46fe54922f6ef"}
Sep 19 08:22:17.008080 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.008049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4bbbc45-66e4-41c2-bb88-859345dfd29c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.008246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.008100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckjr\" (UniqueName: \"kubernetes.io/projected/f4bbbc45-66e4-41c2-bb88-859345dfd29c-kube-api-access-4ckjr\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.008246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.008141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4bbbc45-66e4-41c2-bb88-859345dfd29c-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.008246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.008198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4bbbc45-66e4-41c2-bb88-859345dfd29c-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.008981 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.008962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4bbbc45-66e4-41c2-bb88-859345dfd29c-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.010733 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.010687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4bbbc45-66e4-41c2-bb88-859345dfd29c-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.010840 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.010813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4bbbc45-66e4-41c2-bb88-859345dfd29c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.019024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.019004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckjr\" (UniqueName: \"kubernetes.io/projected/f4bbbc45-66e4-41c2-bb88-859345dfd29c-kube-api-access-4ckjr\") pod \"prometheus-operator-7bd5d56d6-2xwn9\" (UID: \"f4bbbc45-66e4-41c2-bb88-859345dfd29c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.152290 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.152213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845f7b946c-bgn6f"]
Sep 19 08:22:17.172730 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.172705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"
Sep 19 08:22:17.418505 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:17.418431    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b9e6a4_027a_4b92_ae8b_a3df4835678f.slice/crio-e9da26f843cc8b18a1674e74b0ee0336daf912494c97d02044a551cbc5ea6a8c WatchSource:0}: Error finding container e9da26f843cc8b18a1674e74b0ee0336daf912494c97d02044a551cbc5ea6a8c: Status 404 returned error can't find the container with id e9da26f843cc8b18a1674e74b0ee0336daf912494c97d02044a551cbc5ea6a8c
Sep 19 08:22:17.693334 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.693256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9"]
Sep 19 08:22:17.696254 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:17.696225    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bbbc45_66e4_41c2_bb88_859345dfd29c.slice/crio-4042eaa94fac52f39a24d35635ff08aa35c1fde6e88844afc42e3d1a83cf0d50 WatchSource:0}: Error finding container 4042eaa94fac52f39a24d35635ff08aa35c1fde6e88844afc42e3d1a83cf0d50: Status 404 returned error can't find the container with id 4042eaa94fac52f39a24d35635ff08aa35c1fde6e88844afc42e3d1a83cf0d50
Sep 19 08:22:17.970476 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.970375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvd6q" event={"ID":"e58663f3-4077-4718-84d0-016a4127b3f5","Type":"ContainerStarted","Data":"465a8b8dc62f048c4961279a0fcc44d457f1ea62e0953d950ee7c872b3eca5ee"}
Sep 19 08:22:17.972909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.972877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845f7b946c-bgn6f" event={"ID":"89b9e6a4-027a-4b92-ae8b-a3df4835678f","Type":"ContainerStarted","Data":"e9da26f843cc8b18a1674e74b0ee0336daf912494c97d02044a551cbc5ea6a8c"}
Sep 19 08:22:17.976273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.976244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9" event={"ID":"f4bbbc45-66e4-41c2-bb88-859345dfd29c","Type":"ContainerStarted","Data":"4042eaa94fac52f39a24d35635ff08aa35c1fde6e88844afc42e3d1a83cf0d50"}
Sep 19 08:22:17.993919 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:17.993590    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jvd6q" podStartSLOduration=34.096191513 podStartE2EDuration="35.993570836s" totalImagesPullingTime="1.897379323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:21:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:17.991310815 +0000 UTC m=+88.656446385" watchObservedRunningTime="2026-09-19 08:22:17.993570836 +0000 UTC m=+88.658706384"
Sep 19 08:22:21.994741 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:21.994703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9" event={"ID":"f4bbbc45-66e4-41c2-bb88-859345dfd29c","Type":"ContainerStarted","Data":"56b048a0c8af518e79d7a1f719f7c741b4cce3eee3f35648ffdf79f2042b7716"}
Sep 19 08:22:21.995135 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:21.994763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9" event={"ID":"f4bbbc45-66e4-41c2-bb88-859345dfd29c","Type":"ContainerStarted","Data":"a74fde1a4bda0011dc976e4cb81aee30dc47a51b5d6353b932579015e3c3650c"}
Sep 19 08:22:21.996190 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:21.996162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845f7b946c-bgn6f" event={"ID":"89b9e6a4-027a-4b92-ae8b-a3df4835678f","Type":"ContainerStarted","Data":"25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5"}
Sep 19 08:22:22.017564 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:22.017500    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2xwn9" podStartSLOduration=2.1762652989999998 podStartE2EDuration="6.01748145s" totalImagesPullingTime="3.841216151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:22.017385521 +0000 UTC m=+92.682521068" watchObservedRunningTime="2026-09-19 08:22:22.01748145 +0000 UTC m=+92.682616996"
Sep 19 08:22:22.038413 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:22.038368    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-845f7b946c-bgn6f" podStartSLOduration=1.880460173 podStartE2EDuration="6.038354665s" totalImagesPullingTime="4.157894492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:22.037445446 +0000 UTC m=+92.702580992" watchObservedRunningTime="2026-09-19 08:22:22.038354665 +0000 UTC m=+92.703490210"
Sep 19 08:22:23.276996 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.276959    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d4cb58fc9-4q6rg"]
Sep 19 08:22:23.279318 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.279295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.287043 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.287019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 19 08:22:23.293306 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.293283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4cb58fc9-4q6rg"]
Sep 19 08:22:23.358266 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-serving-cert\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.358266 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-service-ca\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.358457 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-config\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.358457 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-oauth-config\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.358457 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-oauth-serving-cert\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.358457 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgx4\" (UniqueName: \"kubernetes.io/projected/8bba66f8-619a-406c-8f1e-cef2ee34aea9-kube-api-access-dxgx4\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.358625 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.358486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-trusted-ca-bundle\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459493 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-oauth-config\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-oauth-serving-cert\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgx4\" (UniqueName: \"kubernetes.io/projected/8bba66f8-619a-406c-8f1e-cef2ee34aea9-kube-api-access-dxgx4\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-trusted-ca-bundle\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-serving-cert\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459850 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-service-ca\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.459850 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.459678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-config\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.460331 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.460307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-service-ca\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.460414 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.460316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-oauth-serving-cert\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.460458 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.460423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-config\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.460514 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.460485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-trusted-ca-bundle\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.462084 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.462064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-serving-cert\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.462168 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.462112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-oauth-config\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.467958 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.467935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgx4\" (UniqueName: \"kubernetes.io/projected/8bba66f8-619a-406c-8f1e-cef2ee34aea9-kube-api-access-dxgx4\") pod \"console-d4cb58fc9-4q6rg\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") " pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.588310 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.588280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:23.873593 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:23.873570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4cb58fc9-4q6rg"]
Sep 19 08:22:23.875418 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:23.875392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bba66f8_619a_406c_8f1e_cef2ee34aea9.slice/crio-cca1e8193582ae6afee0f4c36e25706f2d2bc4942d8aa640aa6b2169d50784ca WatchSource:0}: Error finding container cca1e8193582ae6afee0f4c36e25706f2d2bc4942d8aa640aa6b2169d50784ca: Status 404 returned error can't find the container with id cca1e8193582ae6afee0f4c36e25706f2d2bc4942d8aa640aa6b2169d50784ca
Sep 19 08:22:24.006906 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.006873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4cb58fc9-4q6rg" event={"ID":"8bba66f8-619a-406c-8f1e-cef2ee34aea9","Type":"ContainerStarted","Data":"9274bea74968c228ea8fb79f8c3613e9cbad830d5429812d70d2261b773b8c12"}
Sep 19 08:22:24.007084 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.006928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4cb58fc9-4q6rg" event={"ID":"8bba66f8-619a-406c-8f1e-cef2ee34aea9","Type":"ContainerStarted","Data":"cca1e8193582ae6afee0f4c36e25706f2d2bc4942d8aa640aa6b2169d50784ca"}
Sep 19 08:22:24.032501 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.032449    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d4cb58fc9-4q6rg" podStartSLOduration=1.032434173 podStartE2EDuration="1.032434173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:24.030453736 +0000 UTC m=+94.695589282" watchObservedRunningTime="2026-09-19 08:22:24.032434173 +0000 UTC m=+94.697569719"
Sep 19 08:22:24.395601 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.395568    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fc6gc"]
Sep 19 08:22:24.397823 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.397807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.400162 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.400136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 08:22:24.400509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.400484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 08:22:24.400581 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.400508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 08:22:24.400662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.400648    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wn4fp\""
Sep 19 08:22:24.467232 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-wtmp\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-tls\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c789aa14-3f5d-431e-9888-2a7e9f8e0502-metrics-client-ca\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467369 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-sys\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467558 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467558 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kdk\" (UniqueName: \"kubernetes.io/projected/c789aa14-3f5d-431e-9888-2a7e9f8e0502-kube-api-access-m6kdk\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467558 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-root\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.467558 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.467470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-textfile\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568610 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kdk\" (UniqueName: \"kubernetes.io/projected/c789aa14-3f5d-431e-9888-2a7e9f8e0502-kube-api-access-m6kdk\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568610 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-root\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568798 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-textfile\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568798 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-root\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568798 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-wtmp\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568798 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-tls\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568951 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568951 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c789aa14-3f5d-431e-9888-2a7e9f8e0502-metrics-client-ca\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568951 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-sys\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568951 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.568951 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-wtmp\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.569226 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c789aa14-3f5d-431e-9888-2a7e9f8e0502-sys\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.569226 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.568988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-textfile\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.569496 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.569475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.569580 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.569485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c789aa14-3f5d-431e-9888-2a7e9f8e0502-metrics-client-ca\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.571636 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.571611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-tls\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.572988 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.572967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c789aa14-3f5d-431e-9888-2a7e9f8e0502-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.576865 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.576845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kdk\" (UniqueName: \"kubernetes.io/projected/c789aa14-3f5d-431e-9888-2a7e9f8e0502-kube-api-access-m6kdk\") pod \"node-exporter-fc6gc\" (UID: \"c789aa14-3f5d-431e-9888-2a7e9f8e0502\") " pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.706939 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:24.706871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fc6gc"
Sep 19 08:22:24.716576 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:24.716549    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc789aa14_3f5d_431e_9888_2a7e9f8e0502.slice/crio-7dcea00e1270b656468e27ec24c0f5ed9a534a3d17a492b6ad770456ef43fd99 WatchSource:0}: Error finding container 7dcea00e1270b656468e27ec24c0f5ed9a534a3d17a492b6ad770456ef43fd99: Status 404 returned error can't find the container with id 7dcea00e1270b656468e27ec24c0f5ed9a534a3d17a492b6ad770456ef43fd99
Sep 19 08:22:25.017440 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:25.017356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6gc" event={"ID":"c789aa14-3f5d-431e-9888-2a7e9f8e0502","Type":"ContainerStarted","Data":"7dcea00e1270b656468e27ec24c0f5ed9a534a3d17a492b6ad770456ef43fd99"}
Sep 19 08:22:26.023135 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:26.023083    2579 generic.go:356] "Generic (PLEG): container finished" podID="c789aa14-3f5d-431e-9888-2a7e9f8e0502" containerID="25fee28564d4a66b00b4e570c9b9e529ee0d0710553f6f9b93fc0250aabf0600" exitCode=0
Sep 19 08:22:26.023576 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:26.023154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6gc" event={"ID":"c789aa14-3f5d-431e-9888-2a7e9f8e0502","Type":"ContainerDied","Data":"25fee28564d4a66b00b4e570c9b9e529ee0d0710553f6f9b93fc0250aabf0600"}
Sep 19 08:22:26.872135 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:26.872107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:26.872298 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:26.872172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:26.877037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:26.877015    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:27.029205 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:27.029173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6gc" event={"ID":"c789aa14-3f5d-431e-9888-2a7e9f8e0502","Type":"ContainerStarted","Data":"95364cbe841dccf8644f07fab6663c6bf1df896e408ba78315dc031fa0b071c1"}
Sep 19 08:22:27.029663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:27.029237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6gc" event={"ID":"c789aa14-3f5d-431e-9888-2a7e9f8e0502","Type":"ContainerStarted","Data":"0b867937a84cd8367b7235e80422eb62ef5a9b18cd02a24e61f07bf7d4abe269"}
Sep 19 08:22:27.032987 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:27.032967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:22:27.055730 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:27.055655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fc6gc" podStartSLOduration=2.219251585 podStartE2EDuration="3.055640722s" totalImagesPullingTime="772.907269ms" totalInitContainerRuntime="63.481868ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:27.054131267 +0000 UTC m=+97.719266840" watchObservedRunningTime="2026-09-19 08:22:27.055640722 +0000 UTC m=+97.720776267"
Sep 19 08:22:28.776168 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.776129    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-759cbfbbb-x8bmj"]
Sep 19 08:22:28.778348 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.778327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.785003 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.784978    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 19 08:22:28.785110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.785006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 19 08:22:28.785110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.785068    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 19 08:22:28.785110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.784988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-24svoo999g350\""
Sep 19 08:22:28.785264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.785142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 19 08:22:28.785264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.785147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lh7vb\""
Sep 19 08:22:28.790218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.790190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-759cbfbbb-x8bmj"]
Sep 19 08:22:28.901090 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgqz\" (UniqueName: \"kubernetes.io/projected/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-kube-api-access-mfgqz\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.901211 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-metrics-server-audit-profiles\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.901211 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.901211 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-client-ca-bundle\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.901316 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-audit-log\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.901316 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-secret-metrics-server-tls\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:28.901316 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:28.901303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-secret-metrics-server-client-certs\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002123 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-metrics-server-audit-profiles\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002298 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002298 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-client-ca-bundle\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002298 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-audit-log\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002298 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-secret-metrics-server-tls\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002298 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-secret-metrics-server-client-certs\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002552 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgqz\" (UniqueName: \"kubernetes.io/projected/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-kube-api-access-mfgqz\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002721 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-audit-log\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.002903 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.002879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.003305 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.003285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-metrics-server-audit-profiles\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.004853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.004830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-secret-metrics-server-client-certs\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.005163 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.005144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-client-ca-bundle\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.005222 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.005191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-secret-metrics-server-tls\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.017251 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.017228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgqz\" (UniqueName: \"kubernetes.io/projected/9dcdb463-186f-4c6e-bf9c-896f1a0c8699-kube-api-access-mfgqz\") pod \"metrics-server-759cbfbbb-x8bmj\" (UID: \"9dcdb463-186f-4c6e-bf9c-896f1a0c8699\") " pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.018025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.018008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"]
Sep 19 08:22:29.020737 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.020721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.022878 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.022857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 19 08:22:29.022976 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.022955    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d6rbk\""
Sep 19 08:22:29.033938 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.033919    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"]
Sep 19 08:22:29.093495 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.093471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:29.102783 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.102759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/217d064c-144a-49fc-a902-80736b272668-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jkb4c\" (UID: \"217d064c-144a-49fc-a902-80736b272668\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.102862 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.102847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6pn\" (UniqueName: \"kubernetes.io/projected/217d064c-144a-49fc-a902-80736b272668-kube-api-access-2g6pn\") pod \"monitoring-plugin-5984459ff8-jkb4c\" (UID: \"217d064c-144a-49fc-a902-80736b272668\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.204227 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.204197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6pn\" (UniqueName: \"kubernetes.io/projected/217d064c-144a-49fc-a902-80736b272668-kube-api-access-2g6pn\") pod \"monitoring-plugin-5984459ff8-jkb4c\" (UID: \"217d064c-144a-49fc-a902-80736b272668\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.204371 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.204258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/217d064c-144a-49fc-a902-80736b272668-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jkb4c\" (UID: \"217d064c-144a-49fc-a902-80736b272668\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.206468 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.206440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/217d064c-144a-49fc-a902-80736b272668-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jkb4c\" (UID: \"217d064c-144a-49fc-a902-80736b272668\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.212830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.212801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6pn\" (UniqueName: \"kubernetes.io/projected/217d064c-144a-49fc-a902-80736b272668-kube-api-access-2g6pn\") pod \"monitoring-plugin-5984459ff8-jkb4c\" (UID: \"217d064c-144a-49fc-a902-80736b272668\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.330901 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.330875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:29.373409 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.373379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-759cbfbbb-x8bmj"]
Sep 19 08:22:29.376482 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:29.376446    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcdb463_186f_4c6e_bf9c_896f1a0c8699.slice/crio-1c657c3532498df56fea316920079c0f0ae55afb1462148b775b998f295dd348 WatchSource:0}: Error finding container 1c657c3532498df56fea316920079c0f0ae55afb1462148b775b998f295dd348: Status 404 returned error can't find the container with id 1c657c3532498df56fea316920079c0f0ae55afb1462148b775b998f295dd348
Sep 19 08:22:29.601887 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:29.601852    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217d064c_144a_49fc_a902_80736b272668.slice/crio-45b366435a4574ab54550e376407817bd04466832fa0cab88b45ff378d694b2b WatchSource:0}: Error finding container 45b366435a4574ab54550e376407817bd04466832fa0cab88b45ff378d694b2b: Status 404 returned error can't find the container with id 45b366435a4574ab54550e376407817bd04466832fa0cab88b45ff378d694b2b
Sep 19 08:22:29.604407 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.604370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"]
Sep 19 08:22:29.874596 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:29.874521    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tqgz5"
Sep 19 08:22:30.043498 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.043459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c" event={"ID":"217d064c-144a-49fc-a902-80736b272668","Type":"ContainerStarted","Data":"45b366435a4574ab54550e376407817bd04466832fa0cab88b45ff378d694b2b"}
Sep 19 08:22:30.044597 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.044576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj" event={"ID":"9dcdb463-186f-4c6e-bf9c-896f1a0c8699","Type":"ContainerStarted","Data":"1c657c3532498df56fea316920079c0f0ae55afb1462148b775b998f295dd348"}
Sep 19 08:22:30.525536 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.524920    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:22:30.532541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.532513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.535551 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.535075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 19 08:22:30.535551 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.535414    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cm9qq\""
Sep 19 08:22:30.535807 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.535551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 19 08:22:30.536527 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.536467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 19 08:22:30.536667 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.536650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 19 08:22:30.536971 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.536805    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 19 08:22:30.537560 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.537400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 19 08:22:30.538083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.537748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 19 08:22:30.538083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.537948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 19 08:22:30.538083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.537967    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 19 08:22:30.539091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.538399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 19 08:22:30.539091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.538585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 19 08:22:30.539091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.538620    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-io6md368d3tr\""
Sep 19 08:22:30.539091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.538845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 19 08:22:30.539091 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.538942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 19 08:22:30.544595 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.544455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 19 08:22:30.545618 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.545592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:22:30.614515 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614685 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614685 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614685 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614685 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqm2\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-kube-api-access-mmqm2\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.614909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.614962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615118 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615553 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615553 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615553 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.615553 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.615256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716457 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.716646 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.716976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqm2\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-kube-api-access-mmqm2\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717009 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.717005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717524 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.717033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.717573 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.717554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.718175 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.718149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.718603 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.718583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.719794 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.719767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.720311 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.720287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.720580 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.720559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.722771 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.721843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.722771 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.721889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.722771 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.722453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.724078 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.723376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.724078 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.723541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.724859 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.724460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.724859 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.724475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.724859 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.724806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.725058 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.724993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.725189 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.725171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.726385 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.726319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.727714 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.727673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.729036 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.729008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.731909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.731887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqm2\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-kube-api-access-mmqm2\") pod \"prometheus-k8s-0\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.848078 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.848041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:30.898308 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:30.898274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4cb58fc9-4q6rg"]
Sep 19 08:22:31.574315 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:31.574290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:22:31.576667 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:22:31.576640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8088dc6_8729_41bf_ac5b_ca756d0b2e76.slice/crio-c8fd0c90c04979f4ccdb87514a97c2b551ae95ab2d577b9c108840832dc01dd5 WatchSource:0}: Error finding container c8fd0c90c04979f4ccdb87514a97c2b551ae95ab2d577b9c108840832dc01dd5: Status 404 returned error can't find the container with id c8fd0c90c04979f4ccdb87514a97c2b551ae95ab2d577b9c108840832dc01dd5
Sep 19 08:22:32.053939 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.053903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj" event={"ID":"9dcdb463-186f-4c6e-bf9c-896f1a0c8699","Type":"ContainerStarted","Data":"e2190f874d45788481b86fd4c00874fa04cc6290a262bef314d01fc409d6915a"}
Sep 19 08:22:32.055239 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.055214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"c8fd0c90c04979f4ccdb87514a97c2b551ae95ab2d577b9c108840832dc01dd5"}
Sep 19 08:22:32.056598 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.056576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c" event={"ID":"217d064c-144a-49fc-a902-80736b272668","Type":"ContainerStarted","Data":"45029a403c2c8398afcbd2ee106be938d34b2feceea12a1cc1a504ea9d79c6f3"}
Sep 19 08:22:32.056811 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.056786    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:32.061283 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.061264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c"
Sep 19 08:22:32.071134 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.071089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj" podStartSLOduration=2.148058764 podStartE2EDuration="4.071074311s" totalImagesPullingTime="1.923015547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:32.071014867 +0000 UTC m=+102.736150425" watchObservedRunningTime="2026-09-19 08:22:32.071074311 +0000 UTC m=+102.736209861"
Sep 19 08:22:32.085669 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:32.085633    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jkb4c" podStartSLOduration=1.385202126 podStartE2EDuration="3.085621031s" totalImagesPullingTime="1.700418905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:32.084245191 +0000 UTC m=+102.749380737" watchObservedRunningTime="2026-09-19 08:22:32.085621031 +0000 UTC m=+102.750756616"
Sep 19 08:22:33.062184 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:33.062159    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4" exitCode=0
Sep 19 08:22:33.062468 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:33.062253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"}
Sep 19 08:22:33.589064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:33.589031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:36.078854 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:36.078816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"}
Sep 19 08:22:36.079323 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:36.078906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"}
Sep 19 08:22:38.091765 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:38.091734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"}
Sep 19 08:22:38.092064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:38.091799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"}
Sep 19 08:22:38.092064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:38.091826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"}
Sep 19 08:22:38.092064 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:38.091855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerStarted","Data":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"}
Sep 19 08:22:38.120977 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:38.120932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.169659504 podStartE2EDuration="8.120917024s" totalImagesPullingTime="5.930281025s" totalInitContainerRuntime="20.976495ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:38.118818825 +0000 UTC m=+108.783954371" watchObservedRunningTime="2026-09-19 08:22:38.120917024 +0000 UTC m=+108.786052569"
Sep 19 08:22:40.849112 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:40.849070    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:22:41.741165 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:41.741132    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845f7b946c-bgn6f"]
Sep 19 08:22:49.094323 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:49.094286    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:49.094685 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:49.094342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:22:55.923969 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:55.923886    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d4cb58fc9-4q6rg" podUID="8bba66f8-619a-406c-8f1e-cef2ee34aea9" containerName="console" containerID="cri-o://9274bea74968c228ea8fb79f8c3613e9cbad830d5429812d70d2261b773b8c12" gracePeriod=15
Sep 19 08:22:56.163648 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.163621    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bba66f8-619a-406c-8f1e-cef2ee34aea9" containerID="9274bea74968c228ea8fb79f8c3613e9cbad830d5429812d70d2261b773b8c12" exitCode=0
Sep 19 08:22:56.163789 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.163656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4cb58fc9-4q6rg" event={"ID":"8bba66f8-619a-406c-8f1e-cef2ee34aea9","Type":"ContainerDied","Data":"9274bea74968c228ea8fb79f8c3613e9cbad830d5429812d70d2261b773b8c12"}
Sep 19 08:22:56.163789 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.163733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4cb58fc9-4q6rg" event={"ID":"8bba66f8-619a-406c-8f1e-cef2ee34aea9","Type":"ContainerDied","Data":"cca1e8193582ae6afee0f4c36e25706f2d2bc4942d8aa640aa6b2169d50784ca"}
Sep 19 08:22:56.163789 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.163757    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca1e8193582ae6afee0f4c36e25706f2d2bc4942d8aa640aa6b2169d50784ca"
Sep 19 08:22:56.165760 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.165744    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:56.231837 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.231772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxgx4\" (UniqueName: \"kubernetes.io/projected/8bba66f8-619a-406c-8f1e-cef2ee34aea9-kube-api-access-dxgx4\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.231837 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.231812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-oauth-serving-cert\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.231837 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.231831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-service-ca\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.232086 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.231861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-oauth-config\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.232086 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.231916    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-config\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.232086 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.231953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-trusted-ca-bundle\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.232086 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-serving-cert\") pod \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\" (UID: \"8bba66f8-619a-406c-8f1e-cef2ee34aea9\") "
Sep 19 08:22:56.232286 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:56.232335 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-service-ca" (OuterVolumeSpecName: "service-ca") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:56.232388 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:56.232439 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232384    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-oauth-serving-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:56.232439 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232387    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-config" (OuterVolumeSpecName: "console-config") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:56.232439 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.232404    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-service-ca\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:56.234195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.234177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bba66f8-619a-406c-8f1e-cef2ee34aea9-kube-api-access-dxgx4" (OuterVolumeSpecName: "kube-api-access-dxgx4") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "kube-api-access-dxgx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:56.234292 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.234189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:56.234292 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.234205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8bba66f8-619a-406c-8f1e-cef2ee34aea9" (UID: "8bba66f8-619a-406c-8f1e-cef2ee34aea9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:56.333614 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.333584    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:56.333614 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.333609    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bba66f8-619a-406c-8f1e-cef2ee34aea9-trusted-ca-bundle\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:56.333614 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.333619    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-serving-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:56.333830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.333628    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgx4\" (UniqueName: \"kubernetes.io/projected/8bba66f8-619a-406c-8f1e-cef2ee34aea9-kube-api-access-dxgx4\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:56.333830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:56.333636    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bba66f8-619a-406c-8f1e-cef2ee34aea9-console-oauth-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:22:57.168160 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:57.168129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4cb58fc9-4q6rg"
Sep 19 08:22:57.187945 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:57.187919    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4cb58fc9-4q6rg"]
Sep 19 08:22:57.193926 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:57.193896    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d4cb58fc9-4q6rg"]
Sep 19 08:22:57.603347 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:22:57.603314    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bba66f8-619a-406c-8f1e-cef2ee34aea9" path="/var/lib/kubelet/pods/8bba66f8-619a-406c-8f1e-cef2ee34aea9/volumes"
Sep 19 08:23:06.759822 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:06.759764    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-845f7b946c-bgn6f" podUID="89b9e6a4-027a-4b92-ae8b-a3df4835678f" containerName="console" containerID="cri-o://25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5" gracePeriod=15
Sep 19 08:23:07.022680 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.022617    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:23:07.112851 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.112826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-config\") pod \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") "
Sep 19 08:23:07.113033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.112860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-serving-cert\") pod \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") "
Sep 19 08:23:07.113033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.112900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-service-ca\") pod \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") "
Sep 19 08:23:07.113033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.112918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-oauth-serving-cert\") pod \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") "
Sep 19 08:23:07.113033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.112951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76r9z\" (UniqueName: \"kubernetes.io/projected/89b9e6a4-027a-4b92-ae8b-a3df4835678f-kube-api-access-76r9z\") pod \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") "
Sep 19 08:23:07.113033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.112976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-oauth-config\") pod \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\" (UID: \"89b9e6a4-027a-4b92-ae8b-a3df4835678f\") "
Sep 19 08:23:07.113339 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.113312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-config" (OuterVolumeSpecName: "console-config") pod "89b9e6a4-027a-4b92-ae8b-a3df4835678f" (UID: "89b9e6a4-027a-4b92-ae8b-a3df4835678f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:07.113407 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.113353    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-service-ca" (OuterVolumeSpecName: "service-ca") pod "89b9e6a4-027a-4b92-ae8b-a3df4835678f" (UID: "89b9e6a4-027a-4b92-ae8b-a3df4835678f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:07.113407 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.113374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "89b9e6a4-027a-4b92-ae8b-a3df4835678f" (UID: "89b9e6a4-027a-4b92-ae8b-a3df4835678f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:07.114999 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.114968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "89b9e6a4-027a-4b92-ae8b-a3df4835678f" (UID: "89b9e6a4-027a-4b92-ae8b-a3df4835678f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:07.115099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.115072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b9e6a4-027a-4b92-ae8b-a3df4835678f-kube-api-access-76r9z" (OuterVolumeSpecName: "kube-api-access-76r9z") pod "89b9e6a4-027a-4b92-ae8b-a3df4835678f" (UID: "89b9e6a4-027a-4b92-ae8b-a3df4835678f"). InnerVolumeSpecName "kube-api-access-76r9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:23:07.115139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.115109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "89b9e6a4-027a-4b92-ae8b-a3df4835678f" (UID: "89b9e6a4-027a-4b92-ae8b-a3df4835678f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:07.206381 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.206349    2579 generic.go:356] "Generic (PLEG): container finished" podID="89b9e6a4-027a-4b92-ae8b-a3df4835678f" containerID="25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5" exitCode=0
Sep 19 08:23:07.206509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.206414    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845f7b946c-bgn6f"
Sep 19 08:23:07.206509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.206439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845f7b946c-bgn6f" event={"ID":"89b9e6a4-027a-4b92-ae8b-a3df4835678f","Type":"ContainerDied","Data":"25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5"}
Sep 19 08:23:07.206579 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.206510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845f7b946c-bgn6f" event={"ID":"89b9e6a4-027a-4b92-ae8b-a3df4835678f","Type":"ContainerDied","Data":"e9da26f843cc8b18a1674e74b0ee0336daf912494c97d02044a551cbc5ea6a8c"}
Sep 19 08:23:07.206579 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.206533    2579 scope.go:117] "RemoveContainer" containerID="25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5"
Sep 19 08:23:07.213972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.213954    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-service-ca\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:07.213972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.213973    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-oauth-serving-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:07.214113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.213982    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76r9z\" (UniqueName: \"kubernetes.io/projected/89b9e6a4-027a-4b92-ae8b-a3df4835678f-kube-api-access-76r9z\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:07.214113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.213991    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-oauth-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:07.214113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.213999    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:07.214113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.214007    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89b9e6a4-027a-4b92-ae8b-a3df4835678f-console-serving-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:07.215081 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.215063    2579 scope.go:117] "RemoveContainer" containerID="25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5"
Sep 19 08:23:07.215361 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:07.215342    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5\": container with ID starting with 25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5 not found: ID does not exist" containerID="25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5"
Sep 19 08:23:07.215411 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.215369    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5"} err="failed to get container status \"25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5\": rpc error: code = NotFound desc = could not find container \"25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5\": container with ID starting with 25bd4925ca245f1a90aa091ddb170154b74c2eeff76afde1f55e167ddb7dbda5 not found: ID does not exist"
Sep 19 08:23:07.227083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.227058    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845f7b946c-bgn6f"]
Sep 19 08:23:07.231258 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.231239    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845f7b946c-bgn6f"]
Sep 19 08:23:07.603201 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:07.603170    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b9e6a4-027a-4b92-ae8b-a3df4835678f" path="/var/lib/kubelet/pods/89b9e6a4-027a-4b92-ae8b-a3df4835678f/volumes"
Sep 19 08:23:09.099018 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:09.098991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:23:09.102905 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:09.102886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-759cbfbbb-x8bmj"
Sep 19 08:23:30.848600 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:30.848559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:30.864364 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:30.864337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:31.317589 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:31.317566    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:48.824885 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.824809    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:23:48.825383 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.825094    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="prometheus" containerID="cri-o://454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105" gracePeriod=600
Sep 19 08:23:48.825383 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.825132    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy" containerID="cri-o://ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be" gracePeriod=600
Sep 19 08:23:48.825383 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.825157    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964" gracePeriod=600
Sep 19 08:23:48.825383 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.825177    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="thanos-sidecar" containerID="cri-o://e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470" gracePeriod=600
Sep 19 08:23:48.825383 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.825230    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="config-reloader" containerID="cri-o://ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc" gracePeriod=600
Sep 19 08:23:48.825383 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:48.825180    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-web" containerID="cri-o://a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8" gracePeriod=600
Sep 19 08:23:49.067321 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.067293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.239954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.239868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-web-config\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.239954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.239904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-kube-rbac-proxy\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.239954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.239936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-db\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.239954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.239952    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.239969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-0\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.239987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-serving-certs-ca-bundle\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-trusted-ca-bundle\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-1\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-tls-assets\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240084    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config-out\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-thanos-prometheus-http-client-file\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-metrics-client-certs\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-grpc-tls\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240273 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-metrics-client-ca\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240899 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-tls\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240899 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmqm2\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-kube-api-access-mmqm2\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240899 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-2\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.240899 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-kubelet-serving-ca-bundle\") pod \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\" (UID: \"f8088dc6-8729-41bf-ac5b-ca756d0b2e76\") "
Sep 19 08:23:49.243259 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240351    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.240980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.241025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.241252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.241452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.243294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config" (OuterVolumeSpecName: "config") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.241511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.242927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.243384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.243222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config-out" (OuterVolumeSpecName: "config-out") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:23:49.243812 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.243560    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.243812 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.243656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.243906 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.243822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:23:49.243906 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.243880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:49.244106 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.244076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.244372 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.244344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.245277 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.245251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-kube-api-access-mmqm2" (OuterVolumeSpecName: "kube-api-access-mmqm2") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "kube-api-access-mmqm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:23:49.245445 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.245424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.245609 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.245589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.245675 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.245661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.253617 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.253596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-web-config" (OuterVolumeSpecName: "web-config") pod "f8088dc6-8729-41bf-ac5b-ca756d0b2e76" (UID: "f8088dc6-8729-41bf-ac5b-ca756d0b2e76"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:49.341977 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.341952    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-tls-assets\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.341977 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.341973    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config-out\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.341982    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.341991    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-metrics-client-certs\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342001    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342010    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342018    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-grpc-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342026    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-metrics-client-ca\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342034    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342043    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmqm2\" (UniqueName: \"kubernetes.io/projected/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-kube-api-access-mmqm2\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342051    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342059    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342067    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-web-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342074    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-secret-kube-rbac-proxy\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342082    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-db\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342090    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342099 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342099    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342506 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342107    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342506 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342116    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.342506 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.342124    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8088dc6-8729-41bf-ac5b-ca756d0b2e76-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:23:49.373705 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373670    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964" exitCode=0
Sep 19 08:23:49.373705 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373704    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be" exitCode=0
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373711    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8" exitCode=0
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373716    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470" exitCode=0
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373721    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc" exitCode=0
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"}
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"}
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373789    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"}
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"}
Sep 19 08:23:49.373855 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"}
Sep 19 08:23:49.374327 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373864    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.374327 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373728    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105" exitCode=0
Sep 19 08:23:49.374327 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"}
Sep 19 08:23:49.374327 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.373918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8088dc6-8729-41bf-ac5b-ca756d0b2e76","Type":"ContainerDied","Data":"c8fd0c90c04979f4ccdb87514a97c2b551ae95ab2d577b9c108840832dc01dd5"}
Sep 19 08:23:49.381555 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.381467    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.388127 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.388111    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.394289 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.394266    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.396703 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.396672    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:23:49.400603 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.400583    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:23:49.401805 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.401784    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.408117 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.408101    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.414671 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.414655    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.421133 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.421110    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.421441 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.421423    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.421516 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.421448    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"} err="failed to get container status \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist"
Sep 19 08:23:49.421516 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.421467    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.421778 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.421762    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.421821 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.421793    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"} err="failed to get container status \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist"
Sep 19 08:23:49.421821 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.421808    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.422057 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.422036    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.422155 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422060    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"} err="failed to get container status \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist"
Sep 19 08:23:49.422155 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422075    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.422296 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422280    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:23:49.422344 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.422329    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.422380 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422348    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"} err="failed to get container status \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist"
Sep 19 08:23:49.422380 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422363    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.422560 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422545    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-thanos"
Sep 19 08:23:49.422602 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422564    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-thanos"
Sep 19 08:23:49.422602 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="thanos-sidecar"
Sep 19 08:23:49.422602 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422600    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="thanos-sidecar"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422611    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-web"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-web"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422630    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="config-reloader"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422636    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="config-reloader"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422642    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="prometheus"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422648    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="prometheus"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422654    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422659    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.422586    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422671    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="init-config-reloader"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422681    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="init-config-reloader"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422685    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"} err="failed to get container status \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist"
Sep 19 08:23:49.422718 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422716    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422706    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b9e6a4-027a-4b92-ae8b-a3df4835678f" containerName="console"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422778    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b9e6a4-027a-4b92-ae8b-a3df4835678f" containerName="console"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422793    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bba66f8-619a-406c-8f1e-cef2ee34aea9" containerName="console"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422802    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bba66f8-619a-406c-8f1e-cef2ee34aea9" containerName="console"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422887    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="prometheus"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422901    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b9e6a4-027a-4b92-ae8b-a3df4835678f" containerName="console"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422911    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bba66f8-619a-406c-8f1e-cef2ee34aea9" containerName="console"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422920    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-web"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422929    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422936    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="kube-rbac-proxy-thanos"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422945    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="config-reloader"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422953    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" containerName="thanos-sidecar"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.422972    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.422990    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"} err="failed to get container status \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist"
Sep 19 08:23:49.423152 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423005    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.423670 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:23:49.423232    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.423670 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423254    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"} err="failed to get container status \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist"
Sep 19 08:23:49.423670 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423270    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.423670 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423461    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"} err="failed to get container status \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist"
Sep 19 08:23:49.423670 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423475    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.423843 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423715    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"} err="failed to get container status \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist"
Sep 19 08:23:49.423843 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423729    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.423957 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423936    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"} err="failed to get container status \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist"
Sep 19 08:23:49.424002 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.423960    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.424182 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424158    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"} err="failed to get container status \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist"
Sep 19 08:23:49.424182 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424181    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.424408 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424384    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"} err="failed to get container status \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist"
Sep 19 08:23:49.424452 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424411    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.424655 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424627    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"} err="failed to get container status \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist"
Sep 19 08:23:49.424750 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424655    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.424936 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424916    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"} err="failed to get container status \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist"
Sep 19 08:23:49.425012 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.424933    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.425264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.425226    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"} err="failed to get container status \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist"
Sep 19 08:23:49.425264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.425252    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.425530 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.425508    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"} err="failed to get container status \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist"
Sep 19 08:23:49.425613 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.425531    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.425838 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.425814    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"} err="failed to get container status \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist"
Sep 19 08:23:49.425915 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.425840    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.426105 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426083    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"} err="failed to get container status \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist"
Sep 19 08:23:49.426154 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426107    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.426386 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.426449 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426387    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"} err="failed to get container status \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist"
Sep 19 08:23:49.426449 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426405    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.426906 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426670    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"} err="failed to get container status \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist"
Sep 19 08:23:49.426906 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.426715    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.427028 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427003    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"} err="failed to get container status \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist"
Sep 19 08:23:49.427093 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427032    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.427370 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427347    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"} err="failed to get container status \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist"
Sep 19 08:23:49.427370 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427369    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.427637 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427615    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"} err="failed to get container status \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist"
Sep 19 08:23:49.427684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427640    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.427932 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427913    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"} err="failed to get container status \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist"
Sep 19 08:23:49.428011 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.427934    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.428174 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428157    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"} err="failed to get container status \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist"
Sep 19 08:23:49.428221 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428175    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.428517 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428463    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"} err="failed to get container status \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist"
Sep 19 08:23:49.428579 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428521    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.428791 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428710    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 19 08:23:49.428885 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428798    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"} err="failed to get container status \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist"
Sep 19 08:23:49.428885 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.428819    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.429024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 19 08:23:49.429081 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429052    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 19 08:23:49.429081 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429061    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 19 08:23:49.429176 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429061    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 19 08:23:49.429176 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429114    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 19 08:23:49.429276 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 19 08:23:49.429276 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429200    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"} err="failed to get container status \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist"
Sep 19 08:23:49.429276 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429222    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.429508 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429475    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"} err="failed to get container status \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist"
Sep 19 08:23:49.429508 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429509    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.429671 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429591    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-io6md368d3tr\""
Sep 19 08:23:49.429671 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429661    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cm9qq\""
Sep 19 08:23:49.429806 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 19 08:23:49.429993 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429965    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"} err="failed to get container status \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist"
Sep 19 08:23:49.429993 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.429993    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.430245 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 19 08:23:49.430245 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430243    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 19 08:23:49.430376 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430308    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 19 08:23:49.430376 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430365    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"} err="failed to get container status \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist"
Sep 19 08:23:49.430477 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430380    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.430534 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 19 08:23:49.430806 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430786    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"} err="failed to get container status \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist"
Sep 19 08:23:49.430806 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.430806    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.431139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.431112    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"} err="failed to get container status \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist"
Sep 19 08:23:49.431139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.431138    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.431395 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.431377    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"} err="failed to get container status \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist"
Sep 19 08:23:49.431395 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.431394    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.431761 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.431681    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"} err="failed to get container status \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist"
Sep 19 08:23:49.431761 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.431761    2579 scope.go:117] "RemoveContainer" containerID="f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"
Sep 19 08:23:49.432043 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.432022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964"} err="failed to get container status \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": rpc error: code = NotFound desc = could not find container \"f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964\": container with ID starting with f78c701f8b433d160364ba7fa18dff5549978d1803f40b4a35020d4a1870d964 not found: ID does not exist"
Sep 19 08:23:49.432114 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.432046    2579 scope.go:117] "RemoveContainer" containerID="ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.432294    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be"} err="failed to get container status \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": rpc error: code = NotFound desc = could not find container \"ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be\": container with ID starting with ac208daf67adddfbb84c4894589d825c8c8ffeedaa1bbb707814b161979975be not found: ID does not exist"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.433746    2579 scope.go:117] "RemoveContainer" containerID="a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.434624    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.435304    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8"} err="failed to get container status \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": rpc error: code = NotFound desc = could not find container \"a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8\": container with ID starting with a1b5f507ebcde8bd8ef3921556a95a3a3418facaca550a7d5a684b7958f2e3c8 not found: ID does not exist"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.435332    2579 scope.go:117] "RemoveContainer" containerID="e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.436937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470"} err="failed to get container status \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": rpc error: code = NotFound desc = could not find container \"e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470\": container with ID starting with e6f93511eea2085f1cf072c9c577084caa70f09177b21a4c620611a10d16c470 not found: ID does not exist"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.436962    2579 scope.go:117] "RemoveContainer" containerID="ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"
Sep 19 08:23:49.437788 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.437428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 19 08:23:49.438272 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.438227    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc"} err="failed to get container status \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": rpc error: code = NotFound desc = could not find container \"ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc\": container with ID starting with ed4ae24c2ecd11f01a979425ae2e094704a17bf3a3746e1b85ad856526f3dbbc not found: ID does not exist"
Sep 19 08:23:49.438272 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.438269    2579 scope.go:117] "RemoveContainer" containerID="454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"
Sep 19 08:23:49.438651 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.438625    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105"} err="failed to get container status \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": rpc error: code = NotFound desc = could not find container \"454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105\": container with ID starting with 454e898a5bcba513e7b6e6888dc4a4f9bcdcd623ad40bb0b85defbf4c23f3105 not found: ID does not exist"
Sep 19 08:23:49.438651 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.438646    2579 scope.go:117] "RemoveContainer" containerID="bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"
Sep 19 08:23:49.439402 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.439377    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4"} err="failed to get container status \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": rpc error: code = NotFound desc = could not find container \"bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4\": container with ID starting with bb67e9cb54a028a7fedc8da2114efe00f8c2543c887bea8a92299adc58c9f8a4 not found: ID does not exist"
Sep 19 08:23:49.440474 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.440451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:23:49.542857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.542857 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a4a146e-9478-4dbe-9893-656d7230131d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543037 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.542989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543287 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543287 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543287 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-config\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543287 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a4a146e-9478-4dbe-9893-656d7230131d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543487 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.543679 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.543517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/3a4a146e-9478-4dbe-9893-656d7230131d-kube-api-access-mj4q6\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.602787 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.602761    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8088dc6-8729-41bf-ac5b-ca756d0b2e76" path="/var/lib/kubelet/pods/f8088dc6-8729-41bf-ac5b-ca756d0b2e76/volumes"
Sep 19 08:23:49.643853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.643958 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a4a146e-9478-4dbe-9893-656d7230131d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.643958 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.643958 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.643958 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.643958 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.643998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-config\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a4a146e-9478-4dbe-9893-656d7230131d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/3a4a146e-9478-4dbe-9893-656d7230131d-kube-api-access-mj4q6\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644481 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644949 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.644949 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.644939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.645629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.645599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.647541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.647856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.647875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.647941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.648144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.648283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.648351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648401 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.648392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a4a146e-9478-4dbe-9893-656d7230131d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648869 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.648423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a146e-9478-4dbe-9893-656d7230131d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.648869 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.648446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-config\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.649514 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.649492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a4a146e-9478-4dbe-9893-656d7230131d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.649636 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.649617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.649772 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.649753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.649772 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.649766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.650621 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.650606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a4a146e-9478-4dbe-9893-656d7230131d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.652965 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.652946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/3a4a146e-9478-4dbe-9893-656d7230131d-kube-api-access-mj4q6\") pod \"prometheus-k8s-0\" (UID: \"3a4a146e-9478-4dbe-9893-656d7230131d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:49.743105 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:49.743073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:23:50.019028 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:50.019002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:23:50.020846 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:23:50.020815    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4a146e_9478_4dbe_9893_656d7230131d.slice/crio-5fae48be7012fc34440bdb3e2651884d1931dfc8c6f8ee17720782acf265c330 WatchSource:0}: Error finding container 5fae48be7012fc34440bdb3e2651884d1931dfc8c6f8ee17720782acf265c330: Status 404 returned error can't find the container with id 5fae48be7012fc34440bdb3e2651884d1931dfc8c6f8ee17720782acf265c330
Sep 19 08:23:50.379145 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:50.379073    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a4a146e-9478-4dbe-9893-656d7230131d" containerID="424d17c3ee0b48d6f37e32238f6291f8eadf8d77d29bc583c0bde788443a2e10" exitCode=0
Sep 19 08:23:50.379264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:50.379163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerDied","Data":"424d17c3ee0b48d6f37e32238f6291f8eadf8d77d29bc583c0bde788443a2e10"}
Sep 19 08:23:50.379264 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:50.379233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"5fae48be7012fc34440bdb3e2651884d1931dfc8c6f8ee17720782acf265c330"}
Sep 19 08:23:51.386956 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:51.386919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"9b375fd6725032f6dd9981a0485142a5908ff72f283cc91f7e99a41e2bff5850"}
Sep 19 08:23:51.387416 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:51.386995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"8f18a148e1b8253b4d0d4fae42fb7258a3b377a210c0eba03753c730d7df95ae"}
Sep 19 08:23:51.387416 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:51.387035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"9e30c395cc07cbd91b5c265d16d34de1e059531987fc77ba1f6271a593cce6af"}
Sep 19 08:23:51.387416 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:51.387076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"23f4853ffc3fe6b12e485f52fc8bd7b1f94b02eacf81e22e561fe8ddb0a0941d"}
Sep 19 08:23:51.387416 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:51.387120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"4fb3007fe43ebe3edba2c57a5d109881c3c639657d371afb54ff3a375c0fefa9"}
Sep 19 08:23:51.387416 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:51.387156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a4a146e-9478-4dbe-9893-656d7230131d","Type":"ContainerStarted","Data":"b321e9285aa64c32075de34ec52ee3c3903a96cda4477890b662b51b74701136"}
Sep 19 08:23:54.743804 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:23:54.743758    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:24:49.743644 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:24:49.743609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:24:49.758629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:24:49.758603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:24:49.788871 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:24:49.788812    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=60.770632944 podStartE2EDuration="1m0.788799898s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.166954ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:51.418105431 +0000 UTC m=+182.083240977" watchObservedRunningTime="2026-09-19 08:24:49.788799898 +0000 UTC m=+240.453935444"
Sep 19 08:24:50.625671 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:24:50.625645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:51.482883 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:25:51.482853    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:25:51.486053 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:25:51.486029    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 08:25:51.497851 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:25:51.497829    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:28:51.511002 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:28:51.510972    2579 scope.go:117] "RemoveContainer" containerID="9274bea74968c228ea8fb79f8c3613e9cbad830d5429812d70d2261b773b8c12"
Sep 19 08:30:51.508338 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:30:51.508309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:30:51.519289 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:30:51.519265    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:31:15.536160 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.536128    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"]
Sep 19 08:31:15.539799 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.539778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.543340 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.543317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:31:15.543451 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.543355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 19 08:31:15.543451 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.543420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:31:15.543614 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.543601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b4prd\""
Sep 19 08:31:15.544187 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.544170    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:31:15.544295 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.544281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:31:15.546430 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.546295    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 19 08:31:15.558540 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.558516    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"]
Sep 19 08:31:15.708083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.708046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.708083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.708082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.708338 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.708270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tvq\" (UniqueName: \"kubernetes.io/projected/70d35082-2c8d-41d1-827d-9be3789860b1-kube-api-access-x2tvq\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.708338 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.708296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70d35082-2c8d-41d1-827d-9be3789860b1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.708443 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.708354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809018 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.808980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809018 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvq\" (UniqueName: \"kubernetes.io/projected/70d35082-2c8d-41d1-827d-9be3789860b1-kube-api-access-x2tvq\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70d35082-2c8d-41d1-827d-9be3789860b1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809344 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809423 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:31:15.809404    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:31:15.809505 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:31:15.809493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls podName:70d35082-2c8d-41d1-827d-9be3789860b1 nodeName:}" failed. No retries permitted until 2026-09-19 08:31:16.309470576 +0000 UTC m=+626.974606115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls") pod "isvc-canary-rollback-predictor-7f7b644c9-zjxd6" (UID: "70d35082-2c8d-41d1-827d-9be3789860b1") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:31:15.809579 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70d35082-2c8d-41d1-827d-9be3789860b1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809821 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.809900 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.809831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:15.819621 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:15.819602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tvq\" (UniqueName: \"kubernetes.io/projected/70d35082-2c8d-41d1-827d-9be3789860b1-kube-api-access-x2tvq\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:16.313528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:16.313494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:16.316020 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:16.315995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-zjxd6\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:16.450121 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:16.450086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:16.724151 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:16.724117    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"]
Sep 19 08:31:16.726787 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:31:16.726757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d35082_2c8d_41d1_827d_9be3789860b1.slice/crio-da1f55fff69fdb85eb85773da83a5d7c1a2f1db1dc7847d905712469c267c430 WatchSource:0}: Error finding container da1f55fff69fdb85eb85773da83a5d7c1a2f1db1dc7847d905712469c267c430: Status 404 returned error can't find the container with id da1f55fff69fdb85eb85773da83a5d7c1a2f1db1dc7847d905712469c267c430
Sep 19 08:31:16.729342 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:16.729323    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:31:17.028822 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:17.028742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerStarted","Data":"da1f55fff69fdb85eb85773da83a5d7c1a2f1db1dc7847d905712469c267c430"}
Sep 19 08:31:21.049182 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:21.049136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerStarted","Data":"851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1"}
Sep 19 08:31:25.065450 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:25.065413    2579 generic.go:356] "Generic (PLEG): container finished" podID="70d35082-2c8d-41d1-827d-9be3789860b1" containerID="851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1" exitCode=0
Sep 19 08:31:25.065862 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:25.065485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerDied","Data":"851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1"}
Sep 19 08:31:38.131654 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:38.131624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerStarted","Data":"32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4"}
Sep 19 08:31:41.147932 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:41.147893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerStarted","Data":"bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df"}
Sep 19 08:31:41.148426 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:41.148034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:41.167140 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:41.167088    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podStartSLOduration=3.563266601 podStartE2EDuration="26.167072017s" totalImagesPullingTime="18.502765689s" totalInitContainerRuntime="4.101039727s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:41.165990539 +0000 UTC m=+651.831126085" watchObservedRunningTime="2026-09-19 08:31:41.167072017 +0000 UTC m=+651.832207560"
Sep 19 08:31:42.151333 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:42.151304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:42.152663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:42.152632    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:31:43.155457 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:43.155414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:31:48.160422 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:48.160392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:31:48.160931 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:48.160904    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:31:58.161272 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:31:58.161226    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:32:08.161130 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:32:08.161084    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:32:18.161831 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:32:18.161783    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:32:28.161642 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:32:28.161598    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:32:38.161558 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:32:38.161476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:32:48.161579 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:32:48.161540    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:32:58.161928 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:32:58.161896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:33:21.166735 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.166678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"]
Sep 19 08:33:21.171467 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.171444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.173738 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.173715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-619d8-predictor-serving-cert\""
Sep 19 08:33:21.173884 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.173867    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-619d8-kube-rbac-proxy-sar-config\""
Sep 19 08:33:21.179634 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.179609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"]
Sep 19 08:33:21.264262 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.264220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.264448 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.264272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.264448 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.264382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.264448 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.264417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9z6z\" (UniqueName: \"kubernetes.io/projected/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-api-access-f9z6z\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.264569 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.264454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.365686 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.365646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.365901 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.365720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9z6z\" (UniqueName: \"kubernetes.io/projected/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-api-access-f9z6z\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.365901 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.365763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.365901 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.365804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.365901 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.365833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.366119 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:33:21.365904    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-serving-cert: secret "isvc-xgboost-graph-raw-619d8-predictor-serving-cert" not found
Sep 19 08:33:21.366119 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:33:21.365997    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls podName:2bdb4bf7-2912-4448-9b25-bca2bf627e0c nodeName:}" failed. No retries permitted until 2026-09-19 08:33:21.865974281 +0000 UTC m=+752.531109808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls") pod "isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" (UID: "2bdb4bf7-2912-4448-9b25-bca2bf627e0c") : secret "isvc-xgboost-graph-raw-619d8-predictor-serving-cert" not found
Sep 19 08:33:21.366250 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.366229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.366423 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.366403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.366466 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.366414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.373962 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.373942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9z6z\" (UniqueName: \"kubernetes.io/projected/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-api-access-f9z6z\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.870074 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.870036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:21.872443 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:21.872415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls\") pod \"isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:22.081875 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:22.081833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:22.354804 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:22.354772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"]
Sep 19 08:33:22.357244 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:33:22.357218    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdb4bf7_2912_4448_9b25_bca2bf627e0c.slice/crio-02ba94dc5dde59edafd8a3d08bec1b4f935eaef899034f22c3f863e01984e80e WatchSource:0}: Error finding container 02ba94dc5dde59edafd8a3d08bec1b4f935eaef899034f22c3f863e01984e80e: Status 404 returned error can't find the container with id 02ba94dc5dde59edafd8a3d08bec1b4f935eaef899034f22c3f863e01984e80e
Sep 19 08:33:22.550687 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:22.550638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerStarted","Data":"74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af"}
Sep 19 08:33:22.550893 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:22.550728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerStarted","Data":"02ba94dc5dde59edafd8a3d08bec1b4f935eaef899034f22c3f863e01984e80e"}
Sep 19 08:33:26.567802 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:26.567766    2579 generic.go:356] "Generic (PLEG): container finished" podID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerID="74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af" exitCode=0
Sep 19 08:33:26.568178 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:26.567838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerDied","Data":"74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af"}
Sep 19 08:33:46.661197 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:46.661163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerStarted","Data":"27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25"}
Sep 19 08:33:46.661540 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:46.661231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerStarted","Data":"e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7"}
Sep 19 08:33:46.661540 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:46.661431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:46.679951 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:46.679907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podStartSLOduration=2.010272855 podStartE2EDuration="25.679893123s" totalImagesPullingTime="19.64795576s" totalInitContainerRuntime="4.021664508s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:33:46.679181755 +0000 UTC m=+777.344317303" watchObservedRunningTime="2026-09-19 08:33:46.679893123 +0000 UTC m=+777.345028667"
Sep 19 08:33:47.665515 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:47.665480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:47.666827 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:47.666799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:33:48.669543 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:48.669502    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:33:53.673631 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:53.673604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:33:53.674196 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:33:53.674146    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:34:03.675073 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:03.674994    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:34:13.674925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:13.674885    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:34:23.674878 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:23.674831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:34:33.674083 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:33.674046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:34:43.674820 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:43.674774    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:34:45.214016 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:45.213980    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"]
Sep 19 08:34:45.214522 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:45.214274    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" containerID="cri-o://32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4" gracePeriod=30
Sep 19 08:34:45.214522 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:45.214360    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kube-rbac-proxy" containerID="cri-o://bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df" gracePeriod=30
Sep 19 08:34:45.894441 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:45.894399    2579 generic.go:356] "Generic (PLEG): container finished" podID="70d35082-2c8d-41d1-827d-9be3789860b1" containerID="bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df" exitCode=2
Sep 19 08:34:45.894615 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:45.894451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerDied","Data":"bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df"}
Sep 19 08:34:48.156689 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:48.156646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 19 08:34:48.161076 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:48.161041    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 19 08:34:50.442002 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.441962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"]
Sep 19 08:34:50.444331 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.444313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.446465 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.446442    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 19 08:34:50.446555 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.446442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 19 08:34:50.455600 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.455568    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"]
Sep 19 08:34:50.582900 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.582875    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:34:50.612124 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.612084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f89ebf-5474-45b4-a5d2-6b366d41e293-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.612299 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.612132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdqsp\" (UniqueName: \"kubernetes.io/projected/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-api-access-gdqsp\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.612299 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.612194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.612299 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.612224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f89ebf-5474-45b4-a5d2-6b366d41e293-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.612299 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.612263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.712808 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.712720    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2tvq\" (UniqueName: \"kubernetes.io/projected/70d35082-2c8d-41d1-827d-9be3789860b1-kube-api-access-x2tvq\") pod \"70d35082-2c8d-41d1-827d-9be3789860b1\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") "
Sep 19 08:34:50.712808 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.712786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-cabundle-cert\") pod \"70d35082-2c8d-41d1-827d-9be3789860b1\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") "
Sep 19 08:34:50.713025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.712830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls\") pod \"70d35082-2c8d-41d1-827d-9be3789860b1\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") "
Sep 19 08:34:50.713025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.712889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70d35082-2c8d-41d1-827d-9be3789860b1-kserve-provision-location\") pod \"70d35082-2c8d-41d1-827d-9be3789860b1\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") "
Sep 19 08:34:50.713025 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.712915    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-kube-rbac-proxy-sar-config\") pod \"70d35082-2c8d-41d1-827d-9be3789860b1\" (UID: \"70d35082-2c8d-41d1-827d-9be3789860b1\") "
Sep 19 08:34:50.713213 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.713213 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f89ebf-5474-45b4-a5d2-6b366d41e293-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.713313 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdqsp\" (UniqueName: \"kubernetes.io/projected/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-api-access-gdqsp\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.713313 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.713313 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d35082-2c8d-41d1-827d-9be3789860b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70d35082-2c8d-41d1-827d-9be3789860b1" (UID: "70d35082-2c8d-41d1-827d-9be3789860b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:50.713313 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f89ebf-5474-45b4-a5d2-6b366d41e293-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.713313 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70d35082-2c8d-41d1-827d-9be3789860b1" (UID: "70d35082-2c8d-41d1-827d-9be3789860b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:34:50.713560 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713360    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70d35082-2c8d-41d1-827d-9be3789860b1-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:34:50.713560 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713376    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:34:50.713687 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70d35082-2c8d-41d1-827d-9be3789860b1" (UID: "70d35082-2c8d-41d1-827d-9be3789860b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:34:50.713972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.713972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.713939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f89ebf-5474-45b4-a5d2-6b366d41e293-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.714500 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.714471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.715925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.715896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d35082-2c8d-41d1-827d-9be3789860b1-kube-api-access-x2tvq" (OuterVolumeSpecName: "kube-api-access-x2tvq") pod "70d35082-2c8d-41d1-827d-9be3789860b1" (UID: "70d35082-2c8d-41d1-827d-9be3789860b1"). InnerVolumeSpecName "kube-api-access-x2tvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:34:50.715925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.715912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f89ebf-5474-45b4-a5d2-6b366d41e293-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.716198 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.716179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70d35082-2c8d-41d1-827d-9be3789860b1" (UID: "70d35082-2c8d-41d1-827d-9be3789860b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:34:50.720945 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.720918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdqsp\" (UniqueName: \"kubernetes.io/projected/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-api-access-gdqsp\") pod \"isvc-canary-stop-predictor-5ffdccb94c-6vvmd\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.785015 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.784977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:50.814830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.814790    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70d35082-2c8d-41d1-827d-9be3789860b1-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:34:50.814830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.814820    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70d35082-2c8d-41d1-827d-9be3789860b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:34:50.814830 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.814833    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tvq\" (UniqueName: \"kubernetes.io/projected/70d35082-2c8d-41d1-827d-9be3789860b1-kube-api-access-x2tvq\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:34:50.915972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.915931    2579 generic.go:356] "Generic (PLEG): container finished" podID="70d35082-2c8d-41d1-827d-9be3789860b1" containerID="32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4" exitCode=0
Sep 19 08:34:50.916136 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.915977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerDied","Data":"32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4"}
Sep 19 08:34:50.916136 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.916061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6" event={"ID":"70d35082-2c8d-41d1-827d-9be3789860b1","Type":"ContainerDied","Data":"da1f55fff69fdb85eb85773da83a5d7c1a2f1db1dc7847d905712469c267c430"}
Sep 19 08:34:50.916136 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.915992    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"
Sep 19 08:34:50.916136 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.916104    2579 scope.go:117] "RemoveContainer" containerID="bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df"
Sep 19 08:34:50.925580 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.925561    2579 scope.go:117] "RemoveContainer" containerID="32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4"
Sep 19 08:34:50.933818 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.933798    2579 scope.go:117] "RemoveContainer" containerID="851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1"
Sep 19 08:34:50.939500 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.939468    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"]
Sep 19 08:34:50.942565 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.942536    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-zjxd6"]
Sep 19 08:34:50.942970 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.942951    2579 scope.go:117] "RemoveContainer" containerID="bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df"
Sep 19 08:34:50.943314 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:34:50.943298    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df\": container with ID starting with bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df not found: ID does not exist" containerID="bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df"
Sep 19 08:34:50.943371 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.943323    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df"} err="failed to get container status \"bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df\": rpc error: code = NotFound desc = could not find container \"bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df\": container with ID starting with bde6cd29a6b4c0f6ea520659fb0c0912824ab58937a3ffccfa7b392e03df73df not found: ID does not exist"
Sep 19 08:34:50.943371 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.943343    2579 scope.go:117] "RemoveContainer" containerID="32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4"
Sep 19 08:34:50.943605 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:34:50.943589    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4\": container with ID starting with 32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4 not found: ID does not exist" containerID="32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4"
Sep 19 08:34:50.943648 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.943611    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4"} err="failed to get container status \"32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4\": rpc error: code = NotFound desc = could not find container \"32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4\": container with ID starting with 32cde985ff01f46ade0a37340de9b73a53c1572a2be91f0d5fce2cecdc24a6b4 not found: ID does not exist"
Sep 19 08:34:50.943648 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.943626    2579 scope.go:117] "RemoveContainer" containerID="851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1"
Sep 19 08:34:50.943892 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:34:50.943876    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1\": container with ID starting with 851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1 not found: ID does not exist" containerID="851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1"
Sep 19 08:34:50.943945 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:50.943903    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1"} err="failed to get container status \"851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1\": rpc error: code = NotFound desc = could not find container \"851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1\": container with ID starting with 851da6ccb3af87da53c8e3db99021bbf037cde2fb3aa95b72a94acdf4bfe3dd1 not found: ID does not exist"
Sep 19 08:34:51.057060 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:51.057023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"]
Sep 19 08:34:51.059591 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:34:51.059567    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f89ebf_5474_45b4_a5d2_6b366d41e293.slice/crio-84917714f6529bbeb908420d60639b04b5396e3765b8f6ae7482de97833613a0 WatchSource:0}: Error finding container 84917714f6529bbeb908420d60639b04b5396e3765b8f6ae7482de97833613a0: Status 404 returned error can't find the container with id 84917714f6529bbeb908420d60639b04b5396e3765b8f6ae7482de97833613a0
Sep 19 08:34:51.604668 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:51.604639    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" path="/var/lib/kubelet/pods/70d35082-2c8d-41d1-827d-9be3789860b1/volumes"
Sep 19 08:34:51.921569 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:51.921467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerStarted","Data":"9bb7260524c8fed4465c94b9a1d2f6e40506e4efaa9b580aed6f997dc798b55e"}
Sep 19 08:34:51.921569 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:51.921547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerStarted","Data":"84917714f6529bbeb908420d60639b04b5396e3765b8f6ae7482de97833613a0"}
Sep 19 08:34:53.674798 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:53.674765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:34:54.936492 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:54.936449    2579 generic.go:356] "Generic (PLEG): container finished" podID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerID="9bb7260524c8fed4465c94b9a1d2f6e40506e4efaa9b580aed6f997dc798b55e" exitCode=0
Sep 19 08:34:54.936888 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:54.936531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerDied","Data":"9bb7260524c8fed4465c94b9a1d2f6e40506e4efaa9b580aed6f997dc798b55e"}
Sep 19 08:34:55.943302 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:55.943265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerStarted","Data":"63588093d6827b4d0b0a71b768555eca79d68b38ca1b8027d0c35b37bea9b075"}
Sep 19 08:34:55.943756 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:55.943332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerStarted","Data":"151f9750d071892e2c2b7aca96b0b93a688c39026009611f8706159631cc7bb1"}
Sep 19 08:34:55.943756 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:55.943629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:55.943883 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:55.943801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:34:55.945166 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:55.945140    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:34:55.962205 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:55.962149    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podStartSLOduration=2.205607171 podStartE2EDuration="5.96213325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756526079s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:55.960215972 +0000 UTC m=+846.625351543" watchObservedRunningTime="2026-09-19 08:34:55.96213325 +0000 UTC m=+846.627268795"
Sep 19 08:34:56.948207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:34:56.948161    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:01.952419 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:01.952385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:35:01.952992 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:01.952965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:11.240833 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.240797    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"]
Sep 19 08:35:11.241393 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.241110    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" containerID="cri-o://e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7" gracePeriod=30
Sep 19 08:35:11.241393 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.241170    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kube-rbac-proxy" containerID="cri-o://27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25" gracePeriod=30
Sep 19 08:35:11.429784 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.429751    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"]
Sep 19 08:35:11.430139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430121    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="storage-initializer"
Sep 19 08:35:11.430218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430143    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="storage-initializer"
Sep 19 08:35:11.430218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430158    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container"
Sep 19 08:35:11.430218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430166    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container"
Sep 19 08:35:11.430218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430188    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kube-rbac-proxy"
Sep 19 08:35:11.430218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430197    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kube-rbac-proxy"
Sep 19 08:35:11.430470 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430284    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kserve-container"
Sep 19 08:35:11.430470 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.430297    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70d35082-2c8d-41d1-827d-9be3789860b1" containerName="kube-rbac-proxy"
Sep 19 08:35:11.433402 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.433383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.435608 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.435586    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-caf7a-kube-rbac-proxy-sar-config\""
Sep 19 08:35:11.435722 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.435630    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-caf7a-predictor-serving-cert\""
Sep 19 08:35:11.443439 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.443418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"]
Sep 19 08:35:11.492246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.492175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fdq\" (UniqueName: \"kubernetes.io/projected/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-api-access-72fdq\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.492246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.492212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.492420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.492320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.492420 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.492380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.492509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.492421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593183 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593321 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593321 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fdq\" (UniqueName: \"kubernetes.io/projected/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-api-access-72fdq\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593321 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593321 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593517 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:35:11.593329    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-caf7a-predictor-serving-cert" not found
Sep 19 08:35:11.593517 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:35:11.593416    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls podName:ecdf57ff-1374-4150-9b86-9442bd9fc82d nodeName:}" failed. No retries permitted until 2026-09-19 08:35:12.093393875 +0000 UTC m=+862.758529413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" (UID: "ecdf57ff-1374-4150-9b86-9442bd9fc82d") : secret "isvc-xgboost-graph-raw-hpa-caf7a-predictor-serving-cert" not found
Sep 19 08:35:11.593731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593920 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.593965 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.593903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.601678 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.601652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fdq\" (UniqueName: \"kubernetes.io/projected/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-api-access-72fdq\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:11.952917 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:11.952881    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:12.006218 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:12.006187    2579 generic.go:356] "Generic (PLEG): container finished" podID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerID="27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25" exitCode=2
Sep 19 08:35:12.006356 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:12.006259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerDied","Data":"27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25"}
Sep 19 08:35:12.098106 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:12.098077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:12.100379 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:12.100354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:12.344281 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:12.344245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:12.613983 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:12.613960    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"]
Sep 19 08:35:12.616366 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:35:12.616332    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdf57ff_1374_4150_9b86_9442bd9fc82d.slice/crio-65a5a92b948c6d1e308118f84553d6d2ca4af04a325cc95ce8fefa754f88eae6 WatchSource:0}: Error finding container 65a5a92b948c6d1e308118f84553d6d2ca4af04a325cc95ce8fefa754f88eae6: Status 404 returned error can't find the container with id 65a5a92b948c6d1e308118f84553d6d2ca4af04a325cc95ce8fefa754f88eae6
Sep 19 08:35:13.011767 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:13.011666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerStarted","Data":"fe121468a756a074c2144ed628e0e24014dad0e418ef94ce1d9184648f381837"}
Sep 19 08:35:13.011767 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:13.011756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerStarted","Data":"65a5a92b948c6d1e308118f84553d6d2ca4af04a325cc95ce8fefa754f88eae6"}
Sep 19 08:35:13.670373 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:13.670339    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 19 08:35:13.674432 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:13.674394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 19 08:35:15.387665 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.387638    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:35:15.425961 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.425924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-cabundle-cert\") pod \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") "
Sep 19 08:35:15.425961 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.425965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9z6z\" (UniqueName: \"kubernetes.io/projected/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-api-access-f9z6z\") pod \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") "
Sep 19 08:35:15.426195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.425992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kserve-provision-location\") pod \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") "
Sep 19 08:35:15.426195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.426034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls\") pod \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") "
Sep 19 08:35:15.426195 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.426066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-rbac-proxy-sar-config\") pod \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\" (UID: \"2bdb4bf7-2912-4448-9b25-bca2bf627e0c\") "
Sep 19 08:35:15.426363 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.426330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bdb4bf7-2912-4448-9b25-bca2bf627e0c" (UID: "2bdb4bf7-2912-4448-9b25-bca2bf627e0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:35:15.426363 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.426348    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bdb4bf7-2912-4448-9b25-bca2bf627e0c" (UID: "2bdb4bf7-2912-4448-9b25-bca2bf627e0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:35:15.426553 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.426507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bdb4bf7-2912-4448-9b25-bca2bf627e0c" (UID: "2bdb4bf7-2912-4448-9b25-bca2bf627e0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:35:15.428109 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.428085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bdb4bf7-2912-4448-9b25-bca2bf627e0c" (UID: "2bdb4bf7-2912-4448-9b25-bca2bf627e0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:35:15.428196 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.428137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-api-access-f9z6z" (OuterVolumeSpecName: "kube-api-access-f9z6z") pod "2bdb4bf7-2912-4448-9b25-bca2bf627e0c" (UID: "2bdb4bf7-2912-4448-9b25-bca2bf627e0c"). InnerVolumeSpecName "kube-api-access-f9z6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:35:15.527731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.527624    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:35:15.527731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.527651    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9z6z\" (UniqueName: \"kubernetes.io/projected/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-api-access-f9z6z\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:35:15.527731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.527664    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:35:15.527731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.527674    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:35:15.527731 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:15.527682    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bdb4bf7-2912-4448-9b25-bca2bf627e0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:35:16.026656 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.026625    2579 generic.go:356] "Generic (PLEG): container finished" podID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerID="e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7" exitCode=0
Sep 19 08:35:16.026861 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.026680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"
Sep 19 08:35:16.026861 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.026725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerDied","Data":"e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7"}
Sep 19 08:35:16.026861 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.026787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq" event={"ID":"2bdb4bf7-2912-4448-9b25-bca2bf627e0c","Type":"ContainerDied","Data":"02ba94dc5dde59edafd8a3d08bec1b4f935eaef899034f22c3f863e01984e80e"}
Sep 19 08:35:16.026861 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.026814    2579 scope.go:117] "RemoveContainer" containerID="27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25"
Sep 19 08:35:16.034683 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.034665    2579 scope.go:117] "RemoveContainer" containerID="e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7"
Sep 19 08:35:16.041792 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.041772    2579 scope.go:117] "RemoveContainer" containerID="74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af"
Sep 19 08:35:16.044528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.044508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"]
Sep 19 08:35:16.046653 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.046635    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-619d8-predictor-7d56f4895c-th9tq"]
Sep 19 08:35:16.049683 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.049664    2579 scope.go:117] "RemoveContainer" containerID="27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25"
Sep 19 08:35:16.050028 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:35:16.049989    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25\": container with ID starting with 27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25 not found: ID does not exist" containerID="27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25"
Sep 19 08:35:16.050082 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.050035    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25"} err="failed to get container status \"27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25\": rpc error: code = NotFound desc = could not find container \"27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25\": container with ID starting with 27e945b7797307f1d5502699f6ea2076e4586248b9a431deeb2e7f9f0a0abd25 not found: ID does not exist"
Sep 19 08:35:16.050082 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.050053    2579 scope.go:117] "RemoveContainer" containerID="e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7"
Sep 19 08:35:16.050294 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:35:16.050276    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7\": container with ID starting with e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7 not found: ID does not exist" containerID="e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7"
Sep 19 08:35:16.050330 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.050304    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7"} err="failed to get container status \"e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7\": rpc error: code = NotFound desc = could not find container \"e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7\": container with ID starting with e87c2a93aaa6966d61ca40a794cfb04e2e4dd39066537cc42ebdad2b1bdd18a7 not found: ID does not exist"
Sep 19 08:35:16.050330 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.050327    2579 scope.go:117] "RemoveContainer" containerID="74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af"
Sep 19 08:35:16.050533 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:35:16.050518    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af\": container with ID starting with 74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af not found: ID does not exist" containerID="74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af"
Sep 19 08:35:16.050593 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:16.050535    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af"} err="failed to get container status \"74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af\": rpc error: code = NotFound desc = could not find container \"74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af\": container with ID starting with 74c0a7d8a67518a13608550364bec9064c4f47e06dea6b85181b0bfe62a171af not found: ID does not exist"
Sep 19 08:35:17.033511 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:17.033477    2579 generic.go:356] "Generic (PLEG): container finished" podID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerID="fe121468a756a074c2144ed628e0e24014dad0e418ef94ce1d9184648f381837" exitCode=0
Sep 19 08:35:17.033943 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:17.033525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerDied","Data":"fe121468a756a074c2144ed628e0e24014dad0e418ef94ce1d9184648f381837"}
Sep 19 08:35:17.604019 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:17.603989    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" path="/var/lib/kubelet/pods/2bdb4bf7-2912-4448-9b25-bca2bf627e0c/volumes"
Sep 19 08:35:18.039176 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:18.039101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerStarted","Data":"2c10943fe3083cb11d84cf079cd62b08fed572b37dec9f6857ced82a788707b9"}
Sep 19 08:35:18.039176 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:18.039161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerStarted","Data":"e0caf77cf45bd0a769d97bb0c097d27d9dde5119bcb3e51d45d09a466f6cac2c"}
Sep 19 08:35:18.039545 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:18.039441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:18.039545 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:18.039494    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:18.040963 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:18.040935    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:35:18.057637 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:18.057595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podStartSLOduration=3.245813564 podStartE2EDuration="7.057582505s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.811768941s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:18.05634259 +0000 UTC m=+868.721478137" watchObservedRunningTime="2026-09-19 08:35:18.057582505 +0000 UTC m=+868.722718052"
Sep 19 08:35:19.045267 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:19.045221    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:35:21.953240 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:21.953202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:24.049681 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:24.049653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:35:24.050309 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:24.050283    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:35:31.953703 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:31.953656    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:34.050657 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:34.050567    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:35:41.953434 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:41.953397    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:44.050663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:44.050619    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:35:51.535113 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:51.535068    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:35:51.543577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:51.543554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:35:51.953131 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:51.953094    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:35:54.050508 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:35:54.050468    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:36:01.953808 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:01.953766    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:36:04.050973 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:04.050926    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:36:11.953612 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:11.953567    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 19 08:36:14.050954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:14.050913    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:36:14.599456 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:14.599431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:36:24.051252 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:24.051223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:36:25.460370 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:25.460339    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"]
Sep 19 08:36:25.460877 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:25.460670    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kube-rbac-proxy" containerID="cri-o://63588093d6827b4d0b0a71b768555eca79d68b38ca1b8027d0c35b37bea9b075" gracePeriod=30
Sep 19 08:36:25.460877 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:25.460654    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container" containerID="cri-o://151f9750d071892e2c2b7aca96b0b93a688c39026009611f8706159631cc7bb1" gracePeriod=30
Sep 19 08:36:26.305008 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:26.304976    2579 generic.go:356] "Generic (PLEG): container finished" podID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerID="63588093d6827b4d0b0a71b768555eca79d68b38ca1b8027d0c35b37bea9b075" exitCode=2
Sep 19 08:36:26.305177 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:26.305016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerDied","Data":"63588093d6827b4d0b0a71b768555eca79d68b38ca1b8027d0c35b37bea9b075"}
Sep 19 08:36:26.948469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:26.948415    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 19 08:36:30.323175 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.323143    2579 generic.go:356] "Generic (PLEG): container finished" podID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerID="151f9750d071892e2c2b7aca96b0b93a688c39026009611f8706159631cc7bb1" exitCode=0
Sep 19 08:36:30.323548 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.323205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerDied","Data":"151f9750d071892e2c2b7aca96b0b93a688c39026009611f8706159631cc7bb1"}
Sep 19 08:36:30.397230 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.397206    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:36:30.557974 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.557951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdqsp\" (UniqueName: \"kubernetes.io/projected/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-api-access-gdqsp\") pod \"50f89ebf-5474-45b4-a5d2-6b366d41e293\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") "
Sep 19 08:36:30.558139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f89ebf-5474-45b4-a5d2-6b366d41e293-proxy-tls\") pod \"50f89ebf-5474-45b4-a5d2-6b366d41e293\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") "
Sep 19 08:36:30.558139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-rbac-proxy-sar-config\") pod \"50f89ebf-5474-45b4-a5d2-6b366d41e293\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") "
Sep 19 08:36:30.558139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f89ebf-5474-45b4-a5d2-6b366d41e293-kserve-provision-location\") pod \"50f89ebf-5474-45b4-a5d2-6b366d41e293\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") "
Sep 19 08:36:30.558139 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-cabundle-cert\") pod \"50f89ebf-5474-45b4-a5d2-6b366d41e293\" (UID: \"50f89ebf-5474-45b4-a5d2-6b366d41e293\") "
Sep 19 08:36:30.558414 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f89ebf-5474-45b4-a5d2-6b366d41e293-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50f89ebf-5474-45b4-a5d2-6b366d41e293" (UID: "50f89ebf-5474-45b4-a5d2-6b366d41e293"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:30.558414 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50f89ebf-5474-45b4-a5d2-6b366d41e293" (UID: "50f89ebf-5474-45b4-a5d2-6b366d41e293"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:36:30.558563 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.558520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50f89ebf-5474-45b4-a5d2-6b366d41e293" (UID: "50f89ebf-5474-45b4-a5d2-6b366d41e293"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:36:30.560071 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.560047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-api-access-gdqsp" (OuterVolumeSpecName: "kube-api-access-gdqsp") pod "50f89ebf-5474-45b4-a5d2-6b366d41e293" (UID: "50f89ebf-5474-45b4-a5d2-6b366d41e293"). InnerVolumeSpecName "kube-api-access-gdqsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:36:30.560071 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.560060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f89ebf-5474-45b4-a5d2-6b366d41e293-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50f89ebf-5474-45b4-a5d2-6b366d41e293" (UID: "50f89ebf-5474-45b4-a5d2-6b366d41e293"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:36:30.659311 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.659275    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdqsp\" (UniqueName: \"kubernetes.io/projected/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-api-access-gdqsp\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:36:30.659311 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.659309    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f89ebf-5474-45b4-a5d2-6b366d41e293-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:36:30.659311 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.659319    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:36:30.659537 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.659330    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f89ebf-5474-45b4-a5d2-6b366d41e293-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:36:30.659537 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:30.659340    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f89ebf-5474-45b4-a5d2-6b366d41e293-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:36:31.329319 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.329295    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"
Sep 19 08:36:31.329802 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.329285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd" event={"ID":"50f89ebf-5474-45b4-a5d2-6b366d41e293","Type":"ContainerDied","Data":"84917714f6529bbeb908420d60639b04b5396e3765b8f6ae7482de97833613a0"}
Sep 19 08:36:31.329802 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.329447    2579 scope.go:117] "RemoveContainer" containerID="63588093d6827b4d0b0a71b768555eca79d68b38ca1b8027d0c35b37bea9b075"
Sep 19 08:36:31.337770 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.337751    2579 scope.go:117] "RemoveContainer" containerID="151f9750d071892e2c2b7aca96b0b93a688c39026009611f8706159631cc7bb1"
Sep 19 08:36:31.344860 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.344839    2579 scope.go:117] "RemoveContainer" containerID="9bb7260524c8fed4465c94b9a1d2f6e40506e4efaa9b580aed6f997dc798b55e"
Sep 19 08:36:31.350089 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.350065    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"]
Sep 19 08:36:31.353802 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.353781    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-6vvmd"]
Sep 19 08:36:31.603348 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:36:31.603262    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" path="/var/lib/kubelet/pods/50f89ebf-5474-45b4-a5d2-6b366d41e293/volumes"
Sep 19 08:37:01.548676 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:01.548643    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"]
Sep 19 08:37:01.549156 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:01.548894    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" containerID="cri-o://e0caf77cf45bd0a769d97bb0c097d27d9dde5119bcb3e51d45d09a466f6cac2c" gracePeriod=30
Sep 19 08:37:01.549156 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:01.548927    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kube-rbac-proxy" containerID="cri-o://2c10943fe3083cb11d84cf079cd62b08fed572b37dec9f6857ced82a788707b9" gracePeriod=30
Sep 19 08:37:02.447802 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:02.447766    2579 generic.go:356] "Generic (PLEG): container finished" podID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerID="2c10943fe3083cb11d84cf079cd62b08fed572b37dec9f6857ced82a788707b9" exitCode=2
Sep 19 08:37:02.447967 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:02.447840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerDied","Data":"2c10943fe3083cb11d84cf079cd62b08fed572b37dec9f6857ced82a788707b9"}
Sep 19 08:37:04.045531 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:04.045442    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 19 08:37:04.050823 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:04.050797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 19 08:37:05.462275 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.462242    2579 generic.go:356] "Generic (PLEG): container finished" podID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerID="e0caf77cf45bd0a769d97bb0c097d27d9dde5119bcb3e51d45d09a466f6cac2c" exitCode=0
Sep 19 08:37:05.462616 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.462279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerDied","Data":"e0caf77cf45bd0a769d97bb0c097d27d9dde5119bcb3e51d45d09a466f6cac2c"}
Sep 19 08:37:05.487508 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.487488    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:37:05.552993 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.552955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kserve-provision-location\") pod \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") "
Sep 19 08:37:05.552993 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.552997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72fdq\" (UniqueName: \"kubernetes.io/projected/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-api-access-72fdq\") pod \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") "
Sep 19 08:37:05.553246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.553035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-cabundle-cert\") pod \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") "
Sep 19 08:37:05.553246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.553068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls\") pod \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") "
Sep 19 08:37:05.553246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.553099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-rbac-proxy-sar-config\") pod \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\" (UID: \"ecdf57ff-1374-4150-9b86-9442bd9fc82d\") "
Sep 19 08:37:05.553440 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.553412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecdf57ff-1374-4150-9b86-9442bd9fc82d" (UID: "ecdf57ff-1374-4150-9b86-9442bd9fc82d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:05.553587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.553341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecdf57ff-1374-4150-9b86-9442bd9fc82d" (UID: "ecdf57ff-1374-4150-9b86-9442bd9fc82d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:05.553587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.553520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecdf57ff-1374-4150-9b86-9442bd9fc82d" (UID: "ecdf57ff-1374-4150-9b86-9442bd9fc82d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:05.555085 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.555064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecdf57ff-1374-4150-9b86-9442bd9fc82d" (UID: "ecdf57ff-1374-4150-9b86-9442bd9fc82d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:37:05.555174 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.555127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-api-access-72fdq" (OuterVolumeSpecName: "kube-api-access-72fdq") pod "ecdf57ff-1374-4150-9b86-9442bd9fc82d" (UID: "ecdf57ff-1374-4150-9b86-9442bd9fc82d"). InnerVolumeSpecName "kube-api-access-72fdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:37:05.654174 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.654146    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:37:05.654174 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.654174    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fdq\" (UniqueName: \"kubernetes.io/projected/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-api-access-72fdq\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:37:05.654351 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.654186    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:37:05.654351 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.654198    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecdf57ff-1374-4150-9b86-9442bd9fc82d-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:37:05.654351 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:05.654209    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecdf57ff-1374-4150-9b86-9442bd9fc82d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:37:06.467367 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.467339    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"
Sep 19 08:37:06.467367 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.467357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d" event={"ID":"ecdf57ff-1374-4150-9b86-9442bd9fc82d","Type":"ContainerDied","Data":"65a5a92b948c6d1e308118f84553d6d2ca4af04a325cc95ce8fefa754f88eae6"}
Sep 19 08:37:06.467904 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.467425    2579 scope.go:117] "RemoveContainer" containerID="2c10943fe3083cb11d84cf079cd62b08fed572b37dec9f6857ced82a788707b9"
Sep 19 08:37:06.475233 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.475213    2579 scope.go:117] "RemoveContainer" containerID="e0caf77cf45bd0a769d97bb0c097d27d9dde5119bcb3e51d45d09a466f6cac2c"
Sep 19 08:37:06.482176 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.482153    2579 scope.go:117] "RemoveContainer" containerID="fe121468a756a074c2144ed628e0e24014dad0e418ef94ce1d9184648f381837"
Sep 19 08:37:06.485300 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.485273    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"]
Sep 19 08:37:06.489475 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:06.489454    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-caf7a-predictor-7c9d948f6f-tvk8d"]
Sep 19 08:37:07.602517 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:37:07.602481    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" path="/var/lib/kubelet/pods/ecdf57ff-1374-4150-9b86-9442bd9fc82d/volumes"
Sep 19 08:38:30.867719 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.867666    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"]
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.867970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.867981    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.867996    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="storage-initializer"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868002    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="storage-initializer"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="storage-initializer"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868015    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="storage-initializer"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868022    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868028    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868042    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868053    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="storage-initializer"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868058    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="storage-initializer"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868069    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868075    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868081    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868086    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868092    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868097    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868143    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868151    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868158    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f89ebf-5474-45b4-a5d2-6b366d41e293" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868164    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kserve-container"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868170    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdb4bf7-2912-4448-9b25-bca2bf627e0c" containerName="kube-rbac-proxy"
Sep 19 08:38:30.868203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.868175    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecdf57ff-1374-4150-9b86-9442bd9fc82d" containerName="kube-rbac-proxy"
Sep 19 08:38:30.871260 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.871239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:30.873368 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.873337    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:38:30.873521 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.873350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-53e8e6-kube-rbac-proxy-sar-config\""
Sep 19 08:38:30.873521 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.873413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-53e8e6\""
Sep 19 08:38:30.874168 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.874135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-53e8e6-predictor-serving-cert\""
Sep 19 08:38:30.874168 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.874153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:38:30.874398 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.874200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-53e8e6-dockercfg-2djf6\""
Sep 19 08:38:30.874398 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.874237    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:38:30.881619 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:30.881589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"]
Sep 19 08:38:31.026005 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.025964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.026005 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.026005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-cabundle-cert\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.026220 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.026086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm92s\" (UniqueName: \"kubernetes.io/projected/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-api-access-xm92s\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.026220 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.026120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7449baaf-6b4d-4ad0-b377-4d8d105f1734-proxy-tls\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.026220 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.026142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kserve-provision-location\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.126782 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.126653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7449baaf-6b4d-4ad0-b377-4d8d105f1734-proxy-tls\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.126782 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.126735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kserve-provision-location\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.126782 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.126781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.127100 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.126804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-cabundle-cert\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.127100 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.126867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm92s\" (UniqueName: \"kubernetes.io/projected/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-api-access-xm92s\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.127251 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.127228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kserve-provision-location\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.127550 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.127528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.127594 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.127531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-cabundle-cert\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.129164 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.129147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7449baaf-6b4d-4ad0-b377-4d8d105f1734-proxy-tls\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.135516 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.135492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm92s\" (UniqueName: \"kubernetes.io/projected/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-api-access-xm92s\") pod \"isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") " pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.182237 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.182197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:31.454864 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.454733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"]
Sep 19 08:38:31.457546 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:38:31.457516    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7449baaf_6b4d_4ad0_b377_4d8d105f1734.slice/crio-d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807 WatchSource:0}: Error finding container d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807: Status 404 returned error can't find the container with id d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807
Sep 19 08:38:31.793076 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.793000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" event={"ID":"7449baaf-6b4d-4ad0-b377-4d8d105f1734","Type":"ContainerStarted","Data":"890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac"}
Sep 19 08:38:31.793076 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:31.793071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" event={"ID":"7449baaf-6b4d-4ad0-b377-4d8d105f1734","Type":"ContainerStarted","Data":"d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807"}
Sep 19 08:38:34.811270 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:34.811237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/0.log"
Sep 19 08:38:34.811639 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:34.811278    2579 generic.go:356] "Generic (PLEG): container finished" podID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerID="890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac" exitCode=1
Sep 19 08:38:34.811639 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:34.811312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" event={"ID":"7449baaf-6b4d-4ad0-b377-4d8d105f1734","Type":"ContainerDied","Data":"890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac"}
Sep 19 08:38:35.817245 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:35.817215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/0.log"
Sep 19 08:38:35.817630 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:35.817275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" event={"ID":"7449baaf-6b4d-4ad0-b377-4d8d105f1734","Type":"ContainerStarted","Data":"b9659c89f5cff43bc3c142c66809537950c0c13be4de1bade1610dcb15ee8ffd"}
Sep 19 08:38:39.835649 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.835617    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/1.log"
Sep 19 08:38:39.836092 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.836061    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/0.log"
Sep 19 08:38:39.836134 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.836094    2579 generic.go:356] "Generic (PLEG): container finished" podID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerID="b9659c89f5cff43bc3c142c66809537950c0c13be4de1bade1610dcb15ee8ffd" exitCode=1
Sep 19 08:38:39.836169 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.836134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" event={"ID":"7449baaf-6b4d-4ad0-b377-4d8d105f1734","Type":"ContainerDied","Data":"b9659c89f5cff43bc3c142c66809537950c0c13be4de1bade1610dcb15ee8ffd"}
Sep 19 08:38:39.836245 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.836189    2579 scope.go:117] "RemoveContainer" containerID="890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac"
Sep 19 08:38:39.836560 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.836541    2579 scope.go:117] "RemoveContainer" containerID="890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac"
Sep 19 08:38:39.847260 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:38:39.847235    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_kserve-ci-e2e-test(7449baaf-6b4d-4ad0-b377-4d8d105f1734)\"" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734"
Sep 19 08:38:39.847501 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:38:39.847473    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_kserve-ci-e2e-test_7449baaf-6b4d-4ad0-b377-4d8d105f1734_0 in pod sandbox d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807 from index: no such id: '890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac'" containerID="890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac"
Sep 19 08:38:39.847548 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:39.847513    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_kserve-ci-e2e-test_7449baaf-6b4d-4ad0-b377-4d8d105f1734_0 in pod sandbox d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807 from index: no such id: '890628484f20097c0e176207ae75dcc76d54582b793a1d0aeb5954581846cbac'"
Sep 19 08:38:40.843428 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:40.843394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/1.log"
Sep 19 08:38:46.875505 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:46.875470    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"]
Sep 19 08:38:47.003280 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.003251    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/1.log"
Sep 19 08:38:47.007081 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.007048    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:47.064159 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm92s\" (UniqueName: \"kubernetes.io/projected/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-api-access-xm92s\") pod \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") "
Sep 19 08:38:47.064319 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7449baaf-6b4d-4ad0-b377-4d8d105f1734-proxy-tls\") pod \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") "
Sep 19 08:38:47.064319 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-cabundle-cert\") pod \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") "
Sep 19 08:38:47.064319 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064291    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-rbac-proxy-sar-config\") pod \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") "
Sep 19 08:38:47.064319 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kserve-provision-location\") pod \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\" (UID: \"7449baaf-6b4d-4ad0-b377-4d8d105f1734\") "
Sep 19 08:38:47.064721 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7449baaf-6b4d-4ad0-b377-4d8d105f1734" (UID: "7449baaf-6b4d-4ad0-b377-4d8d105f1734"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:38:47.064813 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7449baaf-6b4d-4ad0-b377-4d8d105f1734" (UID: "7449baaf-6b4d-4ad0-b377-4d8d105f1734"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:38:47.064813 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.064667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7449baaf-6b4d-4ad0-b377-4d8d105f1734" (UID: "7449baaf-6b4d-4ad0-b377-4d8d105f1734"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:47.066474 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.066451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7449baaf-6b4d-4ad0-b377-4d8d105f1734-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7449baaf-6b4d-4ad0-b377-4d8d105f1734" (UID: "7449baaf-6b4d-4ad0-b377-4d8d105f1734"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:38:47.066560 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.066481    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-api-access-xm92s" (OuterVolumeSpecName: "kube-api-access-xm92s") pod "7449baaf-6b4d-4ad0-b377-4d8d105f1734" (UID: "7449baaf-6b4d-4ad0-b377-4d8d105f1734"). InnerVolumeSpecName "kube-api-access-xm92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:38:47.165066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.164976    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.165066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.165004    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.165066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.165014    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm92s\" (UniqueName: \"kubernetes.io/projected/7449baaf-6b4d-4ad0-b377-4d8d105f1734-kube-api-access-xm92s\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.165066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.165025    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7449baaf-6b4d-4ad0-b377-4d8d105f1734-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.165066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.165034    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7449baaf-6b4d-4ad0-b377-4d8d105f1734-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.384687 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.384647    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"]
Sep 19 08:38:47.385032 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.385014    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerName="storage-initializer"
Sep 19 08:38:47.385110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.385036    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerName="storage-initializer"
Sep 19 08:38:47.385110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.385055    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerName="storage-initializer"
Sep 19 08:38:47.385110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.385065    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerName="storage-initializer"
Sep 19 08:38:47.385267 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.385148    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerName="storage-initializer"
Sep 19 08:38:47.385267 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.385159    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" containerName="storage-initializer"
Sep 19 08:38:47.388577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.388551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.390941 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.390914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d1fa7-predictor-serving-cert\""
Sep 19 08:38:47.391073 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.390941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d1fa7-kube-rbac-proxy-sar-config\""
Sep 19 08:38:47.391073 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.390969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b4prd\""
Sep 19 08:38:47.391073 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.390923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:38:47.398790 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.398763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"]
Sep 19 08:38:47.467221 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.467122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.467221 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.467174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23986836-5571-4cfe-9d14-1529cb6cf6fc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.467221 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.467203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.467440 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.467249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9gp\" (UniqueName: \"kubernetes.io/projected/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-api-access-bp9gp\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.467440 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.467281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23986836-5571-4cfe-9d14-1529cb6cf6fc-proxy-tls\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.568317 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.568271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23986836-5571-4cfe-9d14-1529cb6cf6fc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.568317 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.568326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.568591 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.568346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9gp\" (UniqueName: \"kubernetes.io/projected/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-api-access-bp9gp\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.568591 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.568368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23986836-5571-4cfe-9d14-1529cb6cf6fc-proxy-tls\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.568591 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.568429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.568833 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.568802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23986836-5571-4cfe-9d14-1529cb6cf6fc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.569052 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.569031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.569160 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.569142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.571004 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.570981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23986836-5571-4cfe-9d14-1529cb6cf6fc-proxy-tls\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.576821 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.576792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9gp\" (UniqueName: \"kubernetes.io/projected/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-api-access-bp9gp\") pod \"isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.699204 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.699169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:47.873875 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.873850    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4_7449baaf-6b4d-4ad0-b377-4d8d105f1734/storage-initializer/1.log"
Sep 19 08:38:47.874039 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.873897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4" event={"ID":"7449baaf-6b4d-4ad0-b377-4d8d105f1734","Type":"ContainerDied","Data":"d1113eac683c9e25a193d92884c9bb9537f351af5195944b976564af48c75807"}
Sep 19 08:38:47.874039 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.873950    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"
Sep 19 08:38:47.874039 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.873958    2579 scope.go:117] "RemoveContainer" containerID="b9659c89f5cff43bc3c142c66809537950c0c13be4de1bade1610dcb15ee8ffd"
Sep 19 08:38:47.905498 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.905467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"]
Sep 19 08:38:47.909537 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.909506    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53e8e6-predictor-6cbd9985cb-2nnb4"]
Sep 19 08:38:47.971236 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:47.971213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"]
Sep 19 08:38:47.973862 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:38:47.973837    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23986836_5571_4cfe_9d14_1529cb6cf6fc.slice/crio-02a19d349cd4787110219e04a447e16966b06478f1a6ceced43b6b201d2bcd7f WatchSource:0}: Error finding container 02a19d349cd4787110219e04a447e16966b06478f1a6ceced43b6b201d2bcd7f: Status 404 returned error can't find the container with id 02a19d349cd4787110219e04a447e16966b06478f1a6ceced43b6b201d2bcd7f
Sep 19 08:38:48.881015 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:48.880977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerStarted","Data":"aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02"}
Sep 19 08:38:48.881189 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:48.881045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerStarted","Data":"02a19d349cd4787110219e04a447e16966b06478f1a6ceced43b6b201d2bcd7f"}
Sep 19 08:38:49.602927 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:49.602895    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7449baaf-6b4d-4ad0-b377-4d8d105f1734" path="/var/lib/kubelet/pods/7449baaf-6b4d-4ad0-b377-4d8d105f1734/volumes"
Sep 19 08:38:51.893423 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:51.893340    2579 generic.go:356] "Generic (PLEG): container finished" podID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerID="aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02" exitCode=0
Sep 19 08:38:51.893423 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:51.893414    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerDied","Data":"aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02"}
Sep 19 08:38:52.097106 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.097070    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"]
Sep 19 08:38:52.100527 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.100510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.102741 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.102719    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-09a83f-kube-rbac-proxy-sar-config\""
Sep 19 08:38:52.102741 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.102739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-09a83f-predictor-serving-cert\""
Sep 19 08:38:52.102923 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.102722    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-09a83f-dockercfg-v2kd7\""
Sep 19 08:38:52.102923 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.102728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-09a83f\""
Sep 19 08:38:52.110182 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.110158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"]
Sep 19 08:38:52.204555 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.204448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fppb\" (UniqueName: \"kubernetes.io/projected/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-api-access-2fppb\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.204555 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.204506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.204801 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.204616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-cabundle-cert\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.204801 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.204679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ad6014-6645-4a14-8e29-4d335e2d2637-kserve-provision-location\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.204801 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.204750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ad6014-6645-4a14-8e29-4d335e2d2637-proxy-tls\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.305839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.305787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ad6014-6645-4a14-8e29-4d335e2d2637-proxy-tls\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.305839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.305840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fppb\" (UniqueName: \"kubernetes.io/projected/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-api-access-2fppb\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.306090 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.305884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.306090 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.305945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-cabundle-cert\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.306090 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.305995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ad6014-6645-4a14-8e29-4d335e2d2637-kserve-provision-location\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.306412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.306392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ad6014-6645-4a14-8e29-4d335e2d2637-kserve-provision-location\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.306611 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.306594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.306659 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.306601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-cabundle-cert\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.308280 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.308261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ad6014-6645-4a14-8e29-4d335e2d2637-proxy-tls\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.314936 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.314906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fppb\" (UniqueName: \"kubernetes.io/projected/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-api-access-2fppb\") pod \"isvc-init-fail-09a83f-predictor-66bd45f576-rb777\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") " pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.411267 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.411223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:38:52.681890 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.681854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"]
Sep 19 08:38:52.684514 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:38:52.684488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ad6014_6645_4a14_8e29_4d335e2d2637.slice/crio-ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a WatchSource:0}: Error finding container ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a: Status 404 returned error can't find the container with id ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a
Sep 19 08:38:52.906088 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.906038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerStarted","Data":"4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8"}
Sep 19 08:38:52.906551 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.906157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerStarted","Data":"42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0"}
Sep 19 08:38:52.906551 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.906449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:52.906666 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.906613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:52.907715 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.907673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" event={"ID":"d4ad6014-6645-4a14-8e29-4d335e2d2637","Type":"ContainerStarted","Data":"4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508"}
Sep 19 08:38:52.907839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.907732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" event={"ID":"d4ad6014-6645-4a14-8e29-4d335e2d2637","Type":"ContainerStarted","Data":"ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a"}
Sep 19 08:38:52.907839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.907808    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:38:52.924547 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:52.924490    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podStartSLOduration=2.15217745 podStartE2EDuration="5.924472931s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.772295481s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:38:52.924044402 +0000 UTC m=+1083.589179951" watchObservedRunningTime="2026-09-19 08:38:52.924472931 +0000 UTC m=+1083.589608470"
Sep 19 08:38:53.912662 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:53.912618    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:38:57.930515 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:57.930483    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/0.log"
Sep 19 08:38:57.930934 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:57.930525    2579 generic.go:356] "Generic (PLEG): container finished" podID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerID="4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508" exitCode=1
Sep 19 08:38:57.930934 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:57.930555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" event={"ID":"d4ad6014-6645-4a14-8e29-4d335e2d2637","Type":"ContainerDied","Data":"4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508"}
Sep 19 08:38:58.916812 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:58.916780    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:38:58.917391 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:58.917360    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:38:58.937188 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:58.937158    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/0.log"
Sep 19 08:38:58.937613 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:38:58.937235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" event={"ID":"d4ad6014-6645-4a14-8e29-4d335e2d2637","Type":"ContainerStarted","Data":"90d5d8ddff32006ceb66cecbab2b5fc1f6624fc95b4bcad58bb8490b054e8fab"}
Sep 19 08:39:00.947744 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.947712    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/1.log"
Sep 19 08:39:00.948156 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.948139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/0.log"
Sep 19 08:39:00.948216 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.948173    2579 generic.go:356] "Generic (PLEG): container finished" podID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerID="90d5d8ddff32006ceb66cecbab2b5fc1f6624fc95b4bcad58bb8490b054e8fab" exitCode=1
Sep 19 08:39:00.948249 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.948221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" event={"ID":"d4ad6014-6645-4a14-8e29-4d335e2d2637","Type":"ContainerDied","Data":"90d5d8ddff32006ceb66cecbab2b5fc1f6624fc95b4bcad58bb8490b054e8fab"}
Sep 19 08:39:00.948323 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.948312    2579 scope.go:117] "RemoveContainer" containerID="4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508"
Sep 19 08:39:00.948659 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.948636    2579 scope.go:117] "RemoveContainer" containerID="4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508"
Sep 19 08:39:00.958617 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:39:00.958596    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-09a83f-predictor-66bd45f576-rb777_kserve-ci-e2e-test(d4ad6014-6645-4a14-8e29-4d335e2d2637)\"" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637"
Sep 19 08:39:00.958862 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:39:00.958835    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_kserve-ci-e2e-test_d4ad6014-6645-4a14-8e29-4d335e2d2637_0 in pod sandbox ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a from index: no such id: '4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508'" containerID="4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508"
Sep 19 08:39:00.958927 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:00.958869    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_kserve-ci-e2e-test_d4ad6014-6645-4a14-8e29-4d335e2d2637_0 in pod sandbox ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a from index: no such id: '4f3b3c05e53aca27e6bfb7d7f6819cb83d7d89b1f339fbb4cb51f795b8b36508'"
Sep 19 08:39:01.953780 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:01.953755    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/1.log"
Sep 19 08:39:02.072879 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.072848    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"]
Sep 19 08:39:02.203939 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.203878    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/1.log"
Sep 19 08:39:02.207471 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.207449    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:39:02.290562 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-rbac-proxy-sar-config\") pod \"d4ad6014-6645-4a14-8e29-4d335e2d2637\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") "
Sep 19 08:39:02.290770 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290582    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ad6014-6645-4a14-8e29-4d335e2d2637-kserve-provision-location\") pod \"d4ad6014-6645-4a14-8e29-4d335e2d2637\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") "
Sep 19 08:39:02.290770 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fppb\" (UniqueName: \"kubernetes.io/projected/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-api-access-2fppb\") pod \"d4ad6014-6645-4a14-8e29-4d335e2d2637\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") "
Sep 19 08:39:02.290770 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-cabundle-cert\") pod \"d4ad6014-6645-4a14-8e29-4d335e2d2637\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") "
Sep 19 08:39:02.290770 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ad6014-6645-4a14-8e29-4d335e2d2637-proxy-tls\") pod \"d4ad6014-6645-4a14-8e29-4d335e2d2637\" (UID: \"d4ad6014-6645-4a14-8e29-4d335e2d2637\") "
Sep 19 08:39:02.290972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ad6014-6645-4a14-8e29-4d335e2d2637-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4ad6014-6645-4a14-8e29-4d335e2d2637" (UID: "d4ad6014-6645-4a14-8e29-4d335e2d2637"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:39:02.290972 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.290942    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4ad6014-6645-4a14-8e29-4d335e2d2637" (UID: "d4ad6014-6645-4a14-8e29-4d335e2d2637"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:02.291047 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.291024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4ad6014-6645-4a14-8e29-4d335e2d2637" (UID: "d4ad6014-6645-4a14-8e29-4d335e2d2637"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:02.292761 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.292738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ad6014-6645-4a14-8e29-4d335e2d2637-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4ad6014-6645-4a14-8e29-4d335e2d2637" (UID: "d4ad6014-6645-4a14-8e29-4d335e2d2637"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:39:02.292875 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.292816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-api-access-2fppb" (OuterVolumeSpecName: "kube-api-access-2fppb") pod "d4ad6014-6645-4a14-8e29-4d335e2d2637" (UID: "d4ad6014-6645-4a14-8e29-4d335e2d2637"). InnerVolumeSpecName "kube-api-access-2fppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:39:02.392029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.391995    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ad6014-6645-4a14-8e29-4d335e2d2637-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:39:02.392029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.392022    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:39:02.392029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.392031    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ad6014-6645-4a14-8e29-4d335e2d2637-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:39:02.392029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.392042    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fppb\" (UniqueName: \"kubernetes.io/projected/d4ad6014-6645-4a14-8e29-4d335e2d2637-kube-api-access-2fppb\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:39:02.392286 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.392051    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ad6014-6645-4a14-8e29-4d335e2d2637-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:39:02.959869 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.959841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-09a83f-predictor-66bd45f576-rb777_d4ad6014-6645-4a14-8e29-4d335e2d2637/storage-initializer/1.log"
Sep 19 08:39:02.960277 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.959909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777" event={"ID":"d4ad6014-6645-4a14-8e29-4d335e2d2637","Type":"ContainerDied","Data":"ae4dc8119b28cd187133aa131afdf393e1849d92109bf8eaa9235adb8b116b6a"}
Sep 19 08:39:02.960277 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.959931    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"
Sep 19 08:39:02.960277 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.959971    2579 scope.go:117] "RemoveContainer" containerID="90d5d8ddff32006ceb66cecbab2b5fc1f6624fc95b4bcad58bb8490b054e8fab"
Sep 19 08:39:02.995455 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.995422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"]
Sep 19 08:39:02.999717 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:02.999659    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-09a83f-predictor-66bd45f576-rb777"]
Sep 19 08:39:03.607553 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:03.607517    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" path="/var/lib/kubelet/pods/d4ad6014-6645-4a14-8e29-4d335e2d2637/volumes"
Sep 19 08:39:08.918271 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:08.918235    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:39:18.917673 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:18.917622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:39:28.917928 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:28.917891    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:39:38.917488 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:38.917448    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:39:48.917549 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:48.917511    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:39:58.917850 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:39:58.917803    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:40:01.600792 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:01.600748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:40:11.601062 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:11.601026    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:40:21.600926 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:21.600888    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:40:31.600882 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:31.600844    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:40:41.600909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:41.600863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:40:51.559821 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:51.559794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:40:51.565884 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:51.565863    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:40:51.600887 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:40:51.600845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:41:01.601012 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:01.600968    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:41:11.600755 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:11.600717    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:41:19.599013 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:19.598968    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:41:29.603162 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:29.603136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:41:38.530826 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:38.530756    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"]
Sep 19 08:41:38.532995 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:38.530996    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" containerID="cri-o://42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0" gracePeriod=30
Sep 19 08:41:38.532995 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:38.531050    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kube-rbac-proxy" containerID="cri-o://4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8" gracePeriod=30
Sep 19 08:41:38.912709 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:38.912646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 19 08:41:39.553431 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:39.553398    2579 generic.go:356] "Generic (PLEG): container finished" podID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerID="4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8" exitCode=2
Sep 19 08:41:39.553827 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:39.553465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerDied","Data":"4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8"}
Sep 19 08:41:39.599536 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:39.599505    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 19 08:41:43.776123 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776090    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"]
Sep 19 08:41:43.776469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerName="storage-initializer"
Sep 19 08:41:43.776469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776388    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerName="storage-initializer"
Sep 19 08:41:43.776469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776444    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerName="storage-initializer"
Sep 19 08:41:43.776469 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776453    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerName="storage-initializer"
Sep 19 08:41:43.776616 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776523    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerName="storage-initializer"
Sep 19 08:41:43.776616 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.776529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ad6014-6645-4a14-8e29-4d335e2d2637" containerName="storage-initializer"
Sep 19 08:41:43.779631 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.779615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.781723 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.781676    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 19 08:41:43.781835 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.781761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 19 08:41:43.789282 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.789262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"]
Sep 19 08:41:43.839271 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.839240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.839400 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.839296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.839400 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.839317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq47s\" (UniqueName: \"kubernetes.io/projected/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-api-access-zq47s\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.839400 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.839331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc85e1e3-494f-4114-a17b-efb66db0a996-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.839400 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.839348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.912712 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.912661    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 19 08:41:43.940404 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.940366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.940556 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.940432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.940556 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.940453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq47s\" (UniqueName: \"kubernetes.io/projected/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-api-access-zq47s\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.940556 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.940478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc85e1e3-494f-4114-a17b-efb66db0a996-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.940556 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.940494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.940804 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:41:43.940630    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 19 08:41:43.940804 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:41:43.940718    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls podName:fc85e1e3-494f-4114-a17b-efb66db0a996 nodeName:}" failed. No retries permitted until 2026-09-19 08:41:44.440677122 +0000 UTC m=+1255.105812654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls") pod "isvc-canary-create-predictor-76b74f4d6c-nb9vp" (UID: "fc85e1e3-494f-4114-a17b-efb66db0a996") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 19 08:41:43.940912 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.940890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc85e1e3-494f-4114-a17b-efb66db0a996-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.941125 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.941109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.941172 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.941162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:43.951354 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:43.951333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq47s\" (UniqueName: \"kubernetes.io/projected/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-api-access-zq47s\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:44.445372 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:44.445337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:44.447777 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:44.447746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nb9vp\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:44.689648 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:44.689608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:44.961269 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:44.961185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"]
Sep 19 08:41:44.964330 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:41:44.964302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc85e1e3_494f_4114_a17b_efb66db0a996.slice/crio-92df1795061cfd4151c58fc6560ca629d5082e7cb3b34175a2dff24df28ced54 WatchSource:0}: Error finding container 92df1795061cfd4151c58fc6560ca629d5082e7cb3b34175a2dff24df28ced54: Status 404 returned error can't find the container with id 92df1795061cfd4151c58fc6560ca629d5082e7cb3b34175a2dff24df28ced54
Sep 19 08:41:45.579604 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:45.579567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerStarted","Data":"4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410"}
Sep 19 08:41:45.579797 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:45.579623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerStarted","Data":"92df1795061cfd4151c58fc6560ca629d5082e7cb3b34175a2dff24df28ced54"}
Sep 19 08:41:48.768750 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.768724    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:41:48.882990 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.882907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23986836-5571-4cfe-9d14-1529cb6cf6fc-kserve-provision-location\") pod \"23986836-5571-4cfe-9d14-1529cb6cf6fc\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") "
Sep 19 08:41:48.882990 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.882950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp9gp\" (UniqueName: \"kubernetes.io/projected/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-api-access-bp9gp\") pod \"23986836-5571-4cfe-9d14-1529cb6cf6fc\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") "
Sep 19 08:41:48.882990 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.882986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23986836-5571-4cfe-9d14-1529cb6cf6fc-proxy-tls\") pod \"23986836-5571-4cfe-9d14-1529cb6cf6fc\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") "
Sep 19 08:41:48.883541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.883041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-cabundle-cert\") pod \"23986836-5571-4cfe-9d14-1529cb6cf6fc\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") "
Sep 19 08:41:48.883541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.883082    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-rbac-proxy-sar-config\") pod \"23986836-5571-4cfe-9d14-1529cb6cf6fc\" (UID: \"23986836-5571-4cfe-9d14-1529cb6cf6fc\") "
Sep 19 08:41:48.883541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.883206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23986836-5571-4cfe-9d14-1529cb6cf6fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23986836-5571-4cfe-9d14-1529cb6cf6fc" (UID: "23986836-5571-4cfe-9d14-1529cb6cf6fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:41:48.883541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.883280    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23986836-5571-4cfe-9d14-1529cb6cf6fc-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:41:48.883541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.883432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23986836-5571-4cfe-9d14-1529cb6cf6fc" (UID: "23986836-5571-4cfe-9d14-1529cb6cf6fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:41:48.883541 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.883499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23986836-5571-4cfe-9d14-1529cb6cf6fc" (UID: "23986836-5571-4cfe-9d14-1529cb6cf6fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:41:48.885136 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.885102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23986836-5571-4cfe-9d14-1529cb6cf6fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23986836-5571-4cfe-9d14-1529cb6cf6fc" (UID: "23986836-5571-4cfe-9d14-1529cb6cf6fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:41:48.885257 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.885152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-api-access-bp9gp" (OuterVolumeSpecName: "kube-api-access-bp9gp") pod "23986836-5571-4cfe-9d14-1529cb6cf6fc" (UID: "23986836-5571-4cfe-9d14-1529cb6cf6fc"). InnerVolumeSpecName "kube-api-access-bp9gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:41:48.984019 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.983990    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23986836-5571-4cfe-9d14-1529cb6cf6fc-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:41:48.984019 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.984017    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:41:48.984193 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.984026    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:41:48.984193 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:48.984037    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9gp\" (UniqueName: \"kubernetes.io/projected/23986836-5571-4cfe-9d14-1529cb6cf6fc-kube-api-access-bp9gp\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:41:49.597203 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.597169    2579 generic.go:356] "Generic (PLEG): container finished" podID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerID="42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0" exitCode=0
Sep 19 08:41:49.597393 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.597226    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"
Sep 19 08:41:49.597393 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.597254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerDied","Data":"42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0"}
Sep 19 08:41:49.597393 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.597312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l" event={"ID":"23986836-5571-4cfe-9d14-1529cb6cf6fc","Type":"ContainerDied","Data":"02a19d349cd4787110219e04a447e16966b06478f1a6ceced43b6b201d2bcd7f"}
Sep 19 08:41:49.597393 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.597351    2579 scope.go:117] "RemoveContainer" containerID="4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8"
Sep 19 08:41:49.599024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.599001    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerID="4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410" exitCode=0
Sep 19 08:41:49.603050 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.603029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerDied","Data":"4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410"}
Sep 19 08:41:49.606013 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.605991    2579 scope.go:117] "RemoveContainer" containerID="42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0"
Sep 19 08:41:49.613838 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.613823    2579 scope.go:117] "RemoveContainer" containerID="aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02"
Sep 19 08:41:49.631551 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.631537    2579 scope.go:117] "RemoveContainer" containerID="4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8"
Sep 19 08:41:49.631849 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:41:49.631831    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8\": container with ID starting with 4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8 not found: ID does not exist" containerID="4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8"
Sep 19 08:41:49.631941 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.631858    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8"} err="failed to get container status \"4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8\": rpc error: code = NotFound desc = could not find container \"4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8\": container with ID starting with 4581864cc83beeaeaa548c6f945f2637084d74282a09eccbd3e433510d19f5a8 not found: ID does not exist"
Sep 19 08:41:49.631941 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.631876    2579 scope.go:117] "RemoveContainer" containerID="42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0"
Sep 19 08:41:49.632088 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:41:49.632071    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0\": container with ID starting with 42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0 not found: ID does not exist" containerID="42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0"
Sep 19 08:41:49.632144 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.632091    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0"} err="failed to get container status \"42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0\": rpc error: code = NotFound desc = could not find container \"42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0\": container with ID starting with 42fee8d816809b022aa7f7265a97386d59fbb350282bc82e31ec2961d05a48f0 not found: ID does not exist"
Sep 19 08:41:49.632144 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.632103    2579 scope.go:117] "RemoveContainer" containerID="aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02"
Sep 19 08:41:49.632297 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:41:49.632283    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02\": container with ID starting with aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02 not found: ID does not exist" containerID="aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02"
Sep 19 08:41:49.632347 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.632299    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02"} err="failed to get container status \"aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02\": rpc error: code = NotFound desc = could not find container \"aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02\": container with ID starting with aa792f5e2f5a87d951eaa1c52f9c1439a329f16132629bb925f0a0cc6513fd02 not found: ID does not exist"
Sep 19 08:41:49.632400 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.632346    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"]
Sep 19 08:41:49.635779 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:49.635759    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d1fa7-predictor-d45d47c76-kjs7l"]
Sep 19 08:41:50.604566 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:50.604532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerStarted","Data":"af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7"}
Sep 19 08:41:50.605016 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:50.604590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerStarted","Data":"e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b"}
Sep 19 08:41:50.605016 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:50.604888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:50.605016 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:50.604937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:50.606145 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:50.606121    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:41:50.622880 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:50.622833    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podStartSLOduration=3.744483361 podStartE2EDuration="7.622818018s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878334657s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:50.621963258 +0000 UTC m=+1261.287098803" watchObservedRunningTime="2026-09-19 08:41:50.622818018 +0000 UTC m=+1261.287953562"
Sep 19 08:41:51.604947 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:51.604920    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" path="/var/lib/kubelet/pods/23986836-5571-4cfe-9d14-1529cb6cf6fc/volumes"
Sep 19 08:41:51.609489 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:51.609458    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:41:56.614336 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:56.614303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:41:56.614898 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:41:56.614871    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:06.615404 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:06.615364    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:16.615311 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:16.615272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:26.615268 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:26.615226    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:36.614763 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:36.614725    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:46.614909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:46.614863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:56.615059 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:56.615019    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:42:59.598954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:42:59.598914    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 19 08:43:09.602684 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:09.602652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:43:13.792526 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:13.792489    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"]
Sep 19 08:43:13.793015 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:13.792768    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container" containerID="cri-o://e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b" gracePeriod=30
Sep 19 08:43:13.793015 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:13.792820    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kube-rbac-proxy" containerID="cri-o://af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7" gracePeriod=30
Sep 19 08:43:13.922126 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:13.922102    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerID="af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7" exitCode=2
Sep 19 08:43:13.922233 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:13.922168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerDied","Data":"af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7"}
Sep 19 08:43:16.610675 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:16.610632    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 19 08:43:18.427744 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.427720    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:43:18.480820 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.480753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-cabundle-cert\") pod \"fc85e1e3-494f-4114-a17b-efb66db0a996\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") "
Sep 19 08:43:18.480820 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.480806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-rbac-proxy-sar-config\") pod \"fc85e1e3-494f-4114-a17b-efb66db0a996\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") "
Sep 19 08:43:18.481018 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.480836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls\") pod \"fc85e1e3-494f-4114-a17b-efb66db0a996\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") "
Sep 19 08:43:18.481018 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.480859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq47s\" (UniqueName: \"kubernetes.io/projected/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-api-access-zq47s\") pod \"fc85e1e3-494f-4114-a17b-efb66db0a996\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") "
Sep 19 08:43:18.481018 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.480875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc85e1e3-494f-4114-a17b-efb66db0a996-kserve-provision-location\") pod \"fc85e1e3-494f-4114-a17b-efb66db0a996\" (UID: \"fc85e1e3-494f-4114-a17b-efb66db0a996\") "
Sep 19 08:43:18.481172 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.481145    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc85e1e3-494f-4114-a17b-efb66db0a996" (UID: "fc85e1e3-494f-4114-a17b-efb66db0a996"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:18.481228 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.481172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc85e1e3-494f-4114-a17b-efb66db0a996" (UID: "fc85e1e3-494f-4114-a17b-efb66db0a996"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:18.481228 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.481215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc85e1e3-494f-4114-a17b-efb66db0a996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc85e1e3-494f-4114-a17b-efb66db0a996" (UID: "fc85e1e3-494f-4114-a17b-efb66db0a996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:18.482925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.482903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-api-access-zq47s" (OuterVolumeSpecName: "kube-api-access-zq47s") pod "fc85e1e3-494f-4114-a17b-efb66db0a996" (UID: "fc85e1e3-494f-4114-a17b-efb66db0a996"). InnerVolumeSpecName "kube-api-access-zq47s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:18.482925 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.482910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc85e1e3-494f-4114-a17b-efb66db0a996" (UID: "fc85e1e3-494f-4114-a17b-efb66db0a996"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:18.581500 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.581475    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.581500 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.581499    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc85e1e3-494f-4114-a17b-efb66db0a996-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.581660 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.581509    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq47s\" (UniqueName: \"kubernetes.io/projected/fc85e1e3-494f-4114-a17b-efb66db0a996-kube-api-access-zq47s\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.581660 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.581519    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc85e1e3-494f-4114-a17b-efb66db0a996-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.581660 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.581527    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc85e1e3-494f-4114-a17b-efb66db0a996-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.945447 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.945416    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerID="e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b" exitCode=0
Sep 19 08:43:18.945608 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.945484    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"
Sep 19 08:43:18.945608 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.945475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerDied","Data":"e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b"}
Sep 19 08:43:18.945707 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.945624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp" event={"ID":"fc85e1e3-494f-4114-a17b-efb66db0a996","Type":"ContainerDied","Data":"92df1795061cfd4151c58fc6560ca629d5082e7cb3b34175a2dff24df28ced54"}
Sep 19 08:43:18.945707 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.945667    2579 scope.go:117] "RemoveContainer" containerID="af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7"
Sep 19 08:43:18.953564 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.953547    2579 scope.go:117] "RemoveContainer" containerID="e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b"
Sep 19 08:43:18.960954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.960934    2579 scope.go:117] "RemoveContainer" containerID="4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410"
Sep 19 08:43:18.967614 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.967595    2579 scope.go:117] "RemoveContainer" containerID="af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7"
Sep 19 08:43:18.967924 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:43:18.967907    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7\": container with ID starting with af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7 not found: ID does not exist" containerID="af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7"
Sep 19 08:43:18.967982 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.967931    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7"} err="failed to get container status \"af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7\": rpc error: code = NotFound desc = could not find container \"af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7\": container with ID starting with af355810295373eddc672a4a99757dedc134309578832b5c425757053eb074e7 not found: ID does not exist"
Sep 19 08:43:18.967982 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.967948    2579 scope.go:117] "RemoveContainer" containerID="e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b"
Sep 19 08:43:18.968204 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:43:18.968188    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b\": container with ID starting with e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b not found: ID does not exist" containerID="e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b"
Sep 19 08:43:18.968244 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.968209    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b"} err="failed to get container status \"e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b\": rpc error: code = NotFound desc = could not find container \"e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b\": container with ID starting with e5cca83b443f0084713e620167f9172a5567a7035b738b44000b9f0b48fd9a2b not found: ID does not exist"
Sep 19 08:43:18.968244 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.968225    2579 scope.go:117] "RemoveContainer" containerID="4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410"
Sep 19 08:43:18.968473 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:43:18.968447    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410\": container with ID starting with 4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410 not found: ID does not exist" containerID="4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410"
Sep 19 08:43:18.968515 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.968480    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410"} err="failed to get container status \"4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410\": rpc error: code = NotFound desc = could not find container \"4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410\": container with ID starting with 4ef6f0ea53e2429d83840c28a285bf8d8b8175dfb239ba7bb6accde49c262410 not found: ID does not exist"
Sep 19 08:43:18.974938 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.974919    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"]
Sep 19 08:43:18.991523 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:18.991497    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nb9vp"]
Sep 19 08:43:19.065342 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065316    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"]
Sep 19 08:43:19.065598 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065585    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065610    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kube-rbac-proxy"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kube-rbac-proxy"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065627    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kube-rbac-proxy"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065633    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kube-rbac-proxy"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065639    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="storage-initializer"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065658    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="storage-initializer"
Sep 19 08:43:19.065663 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065666    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="storage-initializer"
Sep 19 08:43:19.066024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065672    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="storage-initializer"
Sep 19 08:43:19.066024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065726    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kserve-container"
Sep 19 08:43:19.066024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kserve-container"
Sep 19 08:43:19.066024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065740    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23986836-5571-4cfe-9d14-1529cb6cf6fc" containerName="kube-rbac-proxy"
Sep 19 08:43:19.066024 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.065748    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" containerName="kube-rbac-proxy"
Sep 19 08:43:19.070122 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.070099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.073752 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073602    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 19 08:43:19.073752 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073616    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:43:19.073752 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:43:19.073974 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:43:19.073974 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 19 08:43:19.073974 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073847    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:43:19.073974 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.073855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b4prd\""
Sep 19 08:43:19.081306 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.081287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"]
Sep 19 08:43:19.083992 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.083972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.084074 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.084005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.084122 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.084102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7pv\" (UniqueName: \"kubernetes.io/projected/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-api-access-nc7pv\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.084162 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.084140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.084197 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.084184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.185179 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7pv\" (UniqueName: \"kubernetes.io/projected/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-api-access-nc7pv\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.185270 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.185270 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.185270 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.185386 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.185386 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:43:19.185321    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:43:19.185386 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:43:19.185383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls podName:c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18 nodeName:}" failed. No retries permitted until 2026-09-19 08:43:19.685364823 +0000 UTC m=+1350.350500346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls") pod "isvc-canary-promote-predictor-6546f8697f-m8p7b" (UID: "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:43:19.185670 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.186005 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.186053 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.185988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.195062 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.195041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7pv\" (UniqueName: \"kubernetes.io/projected/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-api-access-nc7pv\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.602616 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.602588    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc85e1e3-494f-4114-a17b-efb66db0a996" path="/var/lib/kubelet/pods/fc85e1e3-494f-4114-a17b-efb66db0a996/volumes"
Sep 19 08:43:19.687526 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.687500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.689793 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.689770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m8p7b\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:19.982865 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:19.982794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:20.252166 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:20.252141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"]
Sep 19 08:43:20.254463 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:43:20.254441    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b69113_e6ef_4bfa_b2b1_29cac4ee4f18.slice/crio-92ad592c4f64834b164a81152ef4a2a6a17a4cd39cde8ac713e027273a427191 WatchSource:0}: Error finding container 92ad592c4f64834b164a81152ef4a2a6a17a4cd39cde8ac713e027273a427191: Status 404 returned error can't find the container with id 92ad592c4f64834b164a81152ef4a2a6a17a4cd39cde8ac713e027273a427191
Sep 19 08:43:20.954629 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:20.954593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerStarted","Data":"bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e"}
Sep 19 08:43:20.955023 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:20.954679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerStarted","Data":"92ad592c4f64834b164a81152ef4a2a6a17a4cd39cde8ac713e027273a427191"}
Sep 19 08:43:24.970665 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:24.970635    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerID="bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e" exitCode=0
Sep 19 08:43:24.971054 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:24.970723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerDied","Data":"bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e"}
Sep 19 08:43:25.976834 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:25.976797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerStarted","Data":"d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6"}
Sep 19 08:43:25.977207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:25.976858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerStarted","Data":"a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9"}
Sep 19 08:43:25.977207 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:25.977043    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:25.997922 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:25.997869    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podStartSLOduration=3.221248444 podStartE2EDuration="6.997855255s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.776606811s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:25.996175647 +0000 UTC m=+1356.661311205" watchObservedRunningTime="2026-09-19 08:43:25.997855255 +0000 UTC m=+1356.662990798"
Sep 19 08:43:26.981303 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:26.981267    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:26.982812 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:26.982782    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:43:27.984848 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:27.984798    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:43:32.989583 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:32.989554    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:43:32.990148 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:32.990121    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:43:42.990828 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:42.990789    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:43:52.990246 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:43:52.990206    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:44:02.990004 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:02.989964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:44:12.990652 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:12.990615    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:44:22.990098 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:22.990055    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:44:32.990387 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:32.990345    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 19 08:44:42.991206 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:42.991134    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:44:54.097422 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.097388    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"]
Sep 19 08:44:54.097965 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.097634    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container" containerID="cri-o://a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9" gracePeriod=30
Sep 19 08:44:54.097965 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.097676    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kube-rbac-proxy" containerID="cri-o://d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6" gracePeriod=30
Sep 19 08:44:54.169433 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.169399    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"]
Sep 19 08:44:54.172831 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.172814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.174888 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.174871    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 19 08:44:54.185443 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.185419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"]
Sep 19 08:44:54.264149 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.264102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.264318 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.264165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795c60cc-0152-4f49-aa2c-ac16f8b46eee-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.264318 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.264210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.264318 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.264270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqks9\" (UniqueName: \"kubernetes.io/projected/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-api-access-mqks9\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.264318 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.264303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.318509 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.318479    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerID="d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6" exitCode=2
Sep 19 08:44:54.318672 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.318556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerDied","Data":"d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6"}
Sep 19 08:44:54.365233 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.365233 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqks9\" (UniqueName: \"kubernetes.io/projected/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-api-access-mqks9\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.365334 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.365334 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.365334 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795c60cc-0152-4f49-aa2c-ac16f8b46eee-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.365609 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.366008 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.365986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.366090 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.366043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.367775 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.367760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795c60cc-0152-4f49-aa2c-ac16f8b46eee-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.372775 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.372750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqks9\" (UniqueName: \"kubernetes.io/projected/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-api-access-mqks9\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.483097 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.483065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:44:54.755269 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:54.755251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"]
Sep 19 08:44:54.758042 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:44:54.758011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795c60cc_0152_4f49_aa2c_ac16f8b46eee.slice/crio-955b503581f1be7248a960d29885154033f26c8a160ea1341bf76575b46a9959 WatchSource:0}: Error finding container 955b503581f1be7248a960d29885154033f26c8a160ea1341bf76575b46a9959: Status 404 returned error can't find the container with id 955b503581f1be7248a960d29885154033f26c8a160ea1341bf76575b46a9959
Sep 19 08:44:55.325449 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:55.325411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerStarted","Data":"a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a"}
Sep 19 08:44:55.325859 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:55.325499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerStarted","Data":"955b503581f1be7248a960d29885154033f26c8a160ea1341bf76575b46a9959"}
Sep 19 08:44:57.985363 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:57.985322    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 19 08:44:59.048054 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.048028    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:44:59.212426 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-cabundle-cert\") pod \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") "
Sep 19 08:44:59.212426 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls\") pod \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") "
Sep 19 08:44:59.212426 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kserve-provision-location\") pod \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") "
Sep 19 08:44:59.212676 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212499    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-rbac-proxy-sar-config\") pod \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") "
Sep 19 08:44:59.212676 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc7pv\" (UniqueName: \"kubernetes.io/projected/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-api-access-nc7pv\") pod \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\" (UID: \"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18\") "
Sep 19 08:44:59.212879 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" (UID: "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:44:59.212879 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" (UID: "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:59.213033 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.212896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" (UID: "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:44:59.214577 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.214556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-api-access-nc7pv" (OuterVolumeSpecName: "kube-api-access-nc7pv") pod "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" (UID: "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18"). InnerVolumeSpecName "kube-api-access-nc7pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:44:59.214650 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.214573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" (UID: "c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:44:59.313442 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.313403    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:44:59.313442 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.313438    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7pv\" (UniqueName: \"kubernetes.io/projected/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kube-api-access-nc7pv\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:44:59.313442 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.313448    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:44:59.313682 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.313458    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:44:59.313682 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.313466    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:44:59.343749 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.343711    2579 generic.go:356] "Generic (PLEG): container finished" podID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerID="a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a" exitCode=0
Sep 19 08:44:59.343927 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.343769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerDied","Data":"a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a"}
Sep 19 08:44:59.345686 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.345666    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerID="a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9" exitCode=0
Sep 19 08:44:59.345777 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.345719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerDied","Data":"a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9"}
Sep 19 08:44:59.345777 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.345739    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"
Sep 19 08:44:59.345777 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.345771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b" event={"ID":"c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18","Type":"ContainerDied","Data":"92ad592c4f64834b164a81152ef4a2a6a17a4cd39cde8ac713e027273a427191"}
Sep 19 08:44:59.345886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.345798    2579 scope.go:117] "RemoveContainer" containerID="d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6"
Sep 19 08:44:59.355856 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.355832    2579 scope.go:117] "RemoveContainer" containerID="a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9"
Sep 19 08:44:59.363847 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.363822    2579 scope.go:117] "RemoveContainer" containerID="bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e"
Sep 19 08:44:59.378528 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.378505    2579 scope.go:117] "RemoveContainer" containerID="d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6"
Sep 19 08:44:59.378856 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:44:59.378837    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6\": container with ID starting with d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6 not found: ID does not exist" containerID="d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6"
Sep 19 08:44:59.378926 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.378865    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6"} err="failed to get container status \"d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6\": rpc error: code = NotFound desc = could not find container \"d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6\": container with ID starting with d7b09e02edda05eeee7b065c92dc566ca66fb9a922ca0c5d0b5b44bad23408a6 not found: ID does not exist"
Sep 19 08:44:59.378926 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.378883    2579 scope.go:117] "RemoveContainer" containerID="a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9"
Sep 19 08:44:59.379379 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:44:59.379344    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9\": container with ID starting with a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9 not found: ID does not exist" containerID="a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9"
Sep 19 08:44:59.379482 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.379383    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9"} err="failed to get container status \"a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9\": rpc error: code = NotFound desc = could not find container \"a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9\": container with ID starting with a3584a88871974a5f136d9cee148d4863db91b3308db99351255cd5fbf8ff3a9 not found: ID does not exist"
Sep 19 08:44:59.379482 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.379407    2579 scope.go:117] "RemoveContainer" containerID="bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e"
Sep 19 08:44:59.379723 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.379616    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"]
Sep 19 08:44:59.379794 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:44:59.379738    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e\": container with ID starting with bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e not found: ID does not exist" containerID="bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e"
Sep 19 08:44:59.379794 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.379765    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e"} err="failed to get container status \"bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e\": rpc error: code = NotFound desc = could not find container \"bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e\": container with ID starting with bdf0939953fd548199e4d9cb0281a6f9fa8fadaddd74e47ee943565c1b76800e not found: ID does not exist"
Sep 19 08:44:59.381713 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.381680    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m8p7b"]
Sep 19 08:44:59.603468 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:44:59.603433    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" path="/var/lib/kubelet/pods/c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18/volumes"
Sep 19 08:45:00.351928 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:00.351893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerStarted","Data":"69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f"}
Sep 19 08:45:00.352410 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:00.351960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerStarted","Data":"e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4"}
Sep 19 08:45:00.352410 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:00.352158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:45:00.373769 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:00.373717    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podStartSLOduration=2.57049596 podStartE2EDuration="6.373688257s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803192297s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:45:00.37230638 +0000 UTC m=+1451.037441926" watchObservedRunningTime="2026-09-19 08:45:00.373688257 +0000 UTC m=+1451.038823802"
Sep 19 08:45:01.356679 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:01.356647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:45:01.358023 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:01.357995    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:02.359783 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:02.359749    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:07.365466 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:07.365430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:45:07.365909 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:07.365883    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:17.366267 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:17.366227    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:27.366309 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:27.366268    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:37.366736 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:37.366681    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:47.366381 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:47.366336    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:45:51.585713 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:51.585663    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:45:51.588953 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:51.588933    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:45:57.365920 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:45:57.365876    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:46:07.365892 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:07.365852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:46:08.599171 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:08.599132    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:46:18.599624 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:18.599588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:46:24.285110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:24.285076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"]
Sep 19 08:46:24.285667 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:24.285365    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" containerID="cri-o://e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4" gracePeriod=30
Sep 19 08:46:24.285667 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:24.285408    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kube-rbac-proxy" containerID="cri-o://69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f" gracePeriod=30
Sep 19 08:46:24.673114 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:24.673085    2579 generic.go:356] "Generic (PLEG): container finished" podID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerID="69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f" exitCode=2
Sep 19 08:46:24.673294 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:24.673147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerDied","Data":"69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f"}
Sep 19 08:46:27.360148 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:27.360107    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 19 08:46:28.599615 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:28.599573    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 19 08:46:28.920828 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:28.920806    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:46:29.004365 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795c60cc-0152-4f49-aa2c-ac16f8b46eee-proxy-tls\") pod \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") "
Sep 19 08:46:29.004365 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-cabundle-cert\") pod \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") "
Sep 19 08:46:29.004588 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kserve-provision-location\") pod \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") "
Sep 19 08:46:29.004588 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqks9\" (UniqueName: \"kubernetes.io/projected/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-api-access-mqks9\") pod \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") "
Sep 19 08:46:29.004588 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-rbac-proxy-sar-config\") pod \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\" (UID: \"795c60cc-0152-4f49-aa2c-ac16f8b46eee\") "
Sep 19 08:46:29.004782 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "795c60cc-0152-4f49-aa2c-ac16f8b46eee" (UID: "795c60cc-0152-4f49-aa2c-ac16f8b46eee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:29.004782 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "795c60cc-0152-4f49-aa2c-ac16f8b46eee" (UID: "795c60cc-0152-4f49-aa2c-ac16f8b46eee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:29.004928 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.004908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "795c60cc-0152-4f49-aa2c-ac16f8b46eee" (UID: "795c60cc-0152-4f49-aa2c-ac16f8b46eee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:29.006426 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.006401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795c60cc-0152-4f49-aa2c-ac16f8b46eee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "795c60cc-0152-4f49-aa2c-ac16f8b46eee" (UID: "795c60cc-0152-4f49-aa2c-ac16f8b46eee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:46:29.006521 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.006491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-api-access-mqks9" (OuterVolumeSpecName: "kube-api-access-mqks9") pod "795c60cc-0152-4f49-aa2c-ac16f8b46eee" (UID: "795c60cc-0152-4f49-aa2c-ac16f8b46eee"). InnerVolumeSpecName "kube-api-access-mqks9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:29.105336 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.105313    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795c60cc-0152-4f49-aa2c-ac16f8b46eee-proxy-tls\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.105437 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.105338    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-cabundle-cert\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.105437 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.105349    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kserve-provision-location\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.105437 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.105358    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqks9\" (UniqueName: \"kubernetes.io/projected/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-api-access-mqks9\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.105437 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.105366    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795c60cc-0152-4f49-aa2c-ac16f8b46eee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.693066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.693035    2579 generic.go:356] "Generic (PLEG): container finished" podID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerID="e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4" exitCode=0
Sep 19 08:46:29.693412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.693083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerDied","Data":"e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4"}
Sep 19 08:46:29.693412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.693100    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"
Sep 19 08:46:29.693412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.693132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm" event={"ID":"795c60cc-0152-4f49-aa2c-ac16f8b46eee","Type":"ContainerDied","Data":"955b503581f1be7248a960d29885154033f26c8a160ea1341bf76575b46a9959"}
Sep 19 08:46:29.693412 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.693158    2579 scope.go:117] "RemoveContainer" containerID="69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f"
Sep 19 08:46:29.701281 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.701266    2579 scope.go:117] "RemoveContainer" containerID="e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4"
Sep 19 08:46:29.710135 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.710110    2579 scope.go:117] "RemoveContainer" containerID="a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a"
Sep 19 08:46:29.710628 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.710605    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"]
Sep 19 08:46:29.714462 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.714439    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-dtgjm"]
Sep 19 08:46:29.717272 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.717257    2579 scope.go:117] "RemoveContainer" containerID="69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f"
Sep 19 08:46:29.717510 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:46:29.717492    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f\": container with ID starting with 69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f not found: ID does not exist" containerID="69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f"
Sep 19 08:46:29.717555 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.717517    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f"} err="failed to get container status \"69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f\": rpc error: code = NotFound desc = could not find container \"69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f\": container with ID starting with 69eb2063a41dff909312d1ba72c6714f52e1809b623266260d52847071b1295f not found: ID does not exist"
Sep 19 08:46:29.717555 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.717532    2579 scope.go:117] "RemoveContainer" containerID="e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4"
Sep 19 08:46:29.717796 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:46:29.717777    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4\": container with ID starting with e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4 not found: ID does not exist" containerID="e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4"
Sep 19 08:46:29.717854 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.717802    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4"} err="failed to get container status \"e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4\": rpc error: code = NotFound desc = could not find container \"e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4\": container with ID starting with e247d90ab87fea843c12c16d086c6adbfab901f79c194cc3bc7003c4fa866fe4 not found: ID does not exist"
Sep 19 08:46:29.717854 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.717820    2579 scope.go:117] "RemoveContainer" containerID="a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a"
Sep 19 08:46:29.718067 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:46:29.718051    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a\": container with ID starting with a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a not found: ID does not exist" containerID="a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a"
Sep 19 08:46:29.718110 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:29.718072    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a"} err="failed to get container status \"a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a\": rpc error: code = NotFound desc = could not find container \"a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a\": container with ID starting with a84a8dd26a83fcfb0b4c66df5efe840a74cd557bac6b3329c528a30b7400851a not found: ID does not exist"
Sep 19 08:46:31.602542 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:46:31.602512    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" path="/var/lib/kubelet/pods/795c60cc-0152-4f49-aa2c-ac16f8b46eee/volumes"
Sep 19 08:48:35.752171 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752138    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dm7p8/must-gather-xtgjm"]
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752402    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kube-rbac-proxy"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752412    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kube-rbac-proxy"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752429    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kube-rbac-proxy"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752435    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kube-rbac-proxy"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752442    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752448    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752474    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="storage-initializer"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752480    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="storage-initializer"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752495    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752508    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="storage-initializer"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752513    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="storage-initializer"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752558    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kube-rbac-proxy"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752566    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kserve-container"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752572    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b69113-e6ef-4bfa-b2b1-29cac4ee4f18" containerName="kserve-container"
Sep 19 08:48:35.752647 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.752578    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="795c60cc-0152-4f49-aa2c-ac16f8b46eee" containerName="kube-rbac-proxy"
Sep 19 08:48:35.755518 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.755503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:35.757534 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.757510    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dm7p8\"/\"openshift-service-ca.crt\""
Sep 19 08:48:35.757677 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.757608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dm7p8\"/\"default-dockercfg-8vc4d\""
Sep 19 08:48:35.757832 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.757813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dm7p8\"/\"kube-root-ca.crt\""
Sep 19 08:48:35.762885 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.762860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0a4cdf7-349a-470e-a391-7dac2534ecde-must-gather-output\") pod \"must-gather-xtgjm\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") " pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:35.763000 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.762906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rgf\" (UniqueName: \"kubernetes.io/projected/a0a4cdf7-349a-470e-a391-7dac2534ecde-kube-api-access-r5rgf\") pod \"must-gather-xtgjm\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") " pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:35.763996 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.763976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dm7p8/must-gather-xtgjm"]
Sep 19 08:48:35.863769 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.863735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0a4cdf7-349a-470e-a391-7dac2534ecde-must-gather-output\") pod \"must-gather-xtgjm\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") " pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:35.863769 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.863767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rgf\" (UniqueName: \"kubernetes.io/projected/a0a4cdf7-349a-470e-a391-7dac2534ecde-kube-api-access-r5rgf\") pod \"must-gather-xtgjm\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") " pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:35.864089 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.864071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0a4cdf7-349a-470e-a391-7dac2534ecde-must-gather-output\") pod \"must-gather-xtgjm\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") " pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:35.871248 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:35.871219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rgf\" (UniqueName: \"kubernetes.io/projected/a0a4cdf7-349a-470e-a391-7dac2534ecde-kube-api-access-r5rgf\") pod \"must-gather-xtgjm\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") " pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:36.064430 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:36.064397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:48:36.330726 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:36.330685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dm7p8/must-gather-xtgjm"]
Sep 19 08:48:36.333294 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:48:36.333270    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a4cdf7_349a_470e_a391_7dac2534ecde.slice/crio-0e3f45d9ba0614768538947b5b86318da26b7a85e2ac2d5326e8ad1fa708252e WatchSource:0}: Error finding container 0e3f45d9ba0614768538947b5b86318da26b7a85e2ac2d5326e8ad1fa708252e: Status 404 returned error can't find the container with id 0e3f45d9ba0614768538947b5b86318da26b7a85e2ac2d5326e8ad1fa708252e
Sep 19 08:48:36.335431 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:36.335415    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:48:37.164933 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:37.164865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm7p8/must-gather-xtgjm" event={"ID":"a0a4cdf7-349a-470e-a391-7dac2534ecde","Type":"ContainerStarted","Data":"0e3f45d9ba0614768538947b5b86318da26b7a85e2ac2d5326e8ad1fa708252e"}
Sep 19 08:48:43.192721 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:43.192666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm7p8/must-gather-xtgjm" event={"ID":"a0a4cdf7-349a-470e-a391-7dac2534ecde","Type":"ContainerStarted","Data":"72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4"}
Sep 19 08:48:43.193123 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:43.192729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm7p8/must-gather-xtgjm" event={"ID":"a0a4cdf7-349a-470e-a391-7dac2534ecde","Type":"ContainerStarted","Data":"41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"}
Sep 19 08:48:43.208340 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:48:43.208288    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dm7p8/must-gather-xtgjm" podStartSLOduration=2.2925393339999998 podStartE2EDuration="8.208270517s" totalImagesPullingTime="5.915731183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:48:43.207384436 +0000 UTC m=+1673.872520000" watchObservedRunningTime="2026-09-19 08:48:43.208270517 +0000 UTC m=+1673.873406061"
Sep 19 08:49:06.287036 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:06.286999    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerID="41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39" exitCode=0
Sep 19 08:49:06.287468 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:06.287067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm7p8/must-gather-xtgjm" event={"ID":"a0a4cdf7-349a-470e-a391-7dac2534ecde","Type":"ContainerDied","Data":"41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"}
Sep 19 08:49:06.287468 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:06.287373    2579 scope.go:117] "RemoveContainer" containerID="41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"
Sep 19 08:49:06.505375 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:06.505348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dm7p8_must-gather-xtgjm_a0a4cdf7-349a-470e-a391-7dac2534ecde/gather/0.log"
Sep 19 08:49:09.903983 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:09.903952    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bx8qb_de943430-0a59-4ea7-b59e-93636d87c4a0/global-pull-secret-syncer/0.log"
Sep 19 08:49:10.050185 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:10.050151    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pwsg6_5c88771f-d05a-4033-bbc7-7c52addcb195/konnectivity-agent/0.log"
Sep 19 08:49:10.098467 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:10.098438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-141.ec2.internal_26cb425dc070dd73c901ca637ab653e6/haproxy/0.log"
Sep 19 08:49:11.911622 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:11.911591    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-dm7p8/must-gather-xtgjm"]
Sep 19 08:49:11.912145 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:11.911762    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-dm7p8/must-gather-xtgjm" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="copy" containerID="cri-o://72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4" gracePeriod=2
Sep 19 08:49:11.918169 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:11.918142    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-dm7p8/must-gather-xtgjm"]
Sep 19 08:49:11.918972 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:49:11.918936    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a0a4cdf7-349a-470e-a391-7dac2534ecde\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-dm7p8\"/\"must-gather-xtgjm\": pods \"must-gather-xtgjm\" not found" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:49:12.136582 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.136563    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dm7p8_must-gather-xtgjm_a0a4cdf7-349a-470e-a391-7dac2534ecde/copy/0.log"
Sep 19 08:49:12.138061 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.138045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:49:12.140439 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:49:12.140410    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xtgjm\" is forbidden: User \"system:node:ip-10-0-140-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm7p8\": no relationship found between node 'ip-10-0-140-141.ec2.internal' and this object" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:49:12.278853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.278785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0a4cdf7-349a-470e-a391-7dac2534ecde-must-gather-output\") pod \"a0a4cdf7-349a-470e-a391-7dac2534ecde\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") "
Sep 19 08:49:12.278853 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.278846    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5rgf\" (UniqueName: \"kubernetes.io/projected/a0a4cdf7-349a-470e-a391-7dac2534ecde-kube-api-access-r5rgf\") pod \"a0a4cdf7-349a-470e-a391-7dac2534ecde\" (UID: \"a0a4cdf7-349a-470e-a391-7dac2534ecde\") "
Sep 19 08:49:12.280744 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.280719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a4cdf7-349a-470e-a391-7dac2534ecde-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a0a4cdf7-349a-470e-a391-7dac2534ecde" (UID: "a0a4cdf7-349a-470e-a391-7dac2534ecde"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:12.280989 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.280972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a4cdf7-349a-470e-a391-7dac2534ecde-kube-api-access-r5rgf" (OuterVolumeSpecName: "kube-api-access-r5rgf") pod "a0a4cdf7-349a-470e-a391-7dac2534ecde" (UID: "a0a4cdf7-349a-470e-a391-7dac2534ecde"). InnerVolumeSpecName "kube-api-access-r5rgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:49:12.313454 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.313435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dm7p8_must-gather-xtgjm_a0a4cdf7-349a-470e-a391-7dac2534ecde/copy/0.log"
Sep 19 08:49:12.313838 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.313819    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerID="72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4" exitCode=143
Sep 19 08:49:12.313908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.313861    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:49:12.313908 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.313884    2579 scope.go:117] "RemoveContainer" containerID="72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4"
Sep 19 08:49:12.317047 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:49:12.317017    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xtgjm\" is forbidden: User \"system:node:ip-10-0-140-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm7p8\": no relationship found between node 'ip-10-0-140-141.ec2.internal' and this object" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:49:12.323935 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.323918    2579 scope.go:117] "RemoveContainer" containerID="41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"
Sep 19 08:49:12.324455 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:49:12.324426    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xtgjm\" is forbidden: User \"system:node:ip-10-0-140-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm7p8\": no relationship found between node 'ip-10-0-140-141.ec2.internal' and this object" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" pod="openshift-must-gather-dm7p8/must-gather-xtgjm"
Sep 19 08:49:12.335841 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.335823    2579 scope.go:117] "RemoveContainer" containerID="72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4"
Sep 19 08:49:12.336099 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:49:12.336080    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4\": container with ID starting with 72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4 not found: ID does not exist" containerID="72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4"
Sep 19 08:49:12.336143 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.336109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4"} err="failed to get container status \"72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4\": rpc error: code = NotFound desc = could not find container \"72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4\": container with ID starting with 72c18adfc49256b19adf6ac1a6e83b09a3f9678892eb77649ebcaa208f8acbd4 not found: ID does not exist"
Sep 19 08:49:12.336143 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.336125    2579 scope.go:117] "RemoveContainer" containerID="41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"
Sep 19 08:49:12.336374 ip-10-0-140-141 kubenswrapper[2579]: E0919 08:49:12.336357    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39\": container with ID starting with 41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39 not found: ID does not exist" containerID="41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"
Sep 19 08:49:12.336424 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.336380    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39"} err="failed to get container status \"41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39\": rpc error: code = NotFound desc = could not find container \"41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39\": container with ID starting with 41a6c005de7fe4c801d62863e7c57347823795f82487efb1d1d8ef055c56ee39 not found: ID does not exist"
Sep 19 08:49:12.379915 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.379893    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rgf\" (UniqueName: \"kubernetes.io/projected/a0a4cdf7-349a-470e-a391-7dac2534ecde-kube-api-access-r5rgf\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:49:12.379915 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:12.379914    2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0a4cdf7-349a-470e-a391-7dac2534ecde-must-gather-output\") on node \"ip-10-0-140-141.ec2.internal\" DevicePath \"\""
Sep 19 08:49:13.603143 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:13.603115    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" path="/var/lib/kubelet/pods/a0a4cdf7-349a-470e-a391-7dac2534ecde/volumes"
Sep 19 08:49:14.564324 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:14.564299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-x7s9q_07e0274e-0b96-4df0-a08e-6c1fe4ea9172/cluster-monitoring-operator/0.log"
Sep 19 08:49:14.672029 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:14.672006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-759cbfbbb-x8bmj_9dcdb463-186f-4c6e-bf9c-896f1a0c8699/metrics-server/0.log"
Sep 19 08:49:14.699643 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:14.699621    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-jkb4c_217d064c-144a-49fc-a902-80736b272668/monitoring-plugin/0.log"
Sep 19 08:49:14.729282 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:14.729259    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc6gc_c789aa14-3f5d-431e-9888-2a7e9f8e0502/node-exporter/0.log"
Sep 19 08:49:14.750783 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:14.750761    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc6gc_c789aa14-3f5d-431e-9888-2a7e9f8e0502/kube-rbac-proxy/0.log"
Sep 19 08:49:14.790085 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:14.790065    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc6gc_c789aa14-3f5d-431e-9888-2a7e9f8e0502/init-textfile/0.log"
Sep 19 08:49:15.093944 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.093919    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/prometheus/0.log"
Sep 19 08:49:15.117346 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.117324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/config-reloader/0.log"
Sep 19 08:49:15.138389 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.138370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/thanos-sidecar/0.log"
Sep 19 08:49:15.162799 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.162782    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/kube-rbac-proxy-web/0.log"
Sep 19 08:49:15.182945 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.182927    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/kube-rbac-proxy/0.log"
Sep 19 08:49:15.203053 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.203035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/kube-rbac-proxy-thanos/0.log"
Sep 19 08:49:15.223522 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.223505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a4a146e-9478-4dbe-9893-656d7230131d/init-config-reloader/0.log"
Sep 19 08:49:15.253819 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.253801    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-2xwn9_f4bbbc45-66e4-41c2-bb88-859345dfd29c/prometheus-operator/0.log"
Sep 19 08:49:15.277225 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.277206    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-2xwn9_f4bbbc45-66e4-41c2-bb88-859345dfd29c/kube-rbac-proxy/0.log"
Sep 19 08:49:15.305016 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:15.304998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-7xvfv_33c79094-d1a0-4861-a718-5cbfbb580b66/prometheus-operator-admission-webhook/0.log"
Sep 19 08:49:16.881324 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:16.881302    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-c8ln2_5eed00ec-765a-4ff5-ada9-02072cf59fb3/networking-console-plugin/0.log"
Sep 19 08:49:18.340415 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340384    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"]
Sep 19 08:49:18.340886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340720    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="copy"
Sep 19 08:49:18.340886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340733    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="copy"
Sep 19 08:49:18.340886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340747    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="gather"
Sep 19 08:49:18.340886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340753    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="gather"
Sep 19 08:49:18.340886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340804    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="gather"
Sep 19 08:49:18.340886 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.340813    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a4cdf7-349a-470e-a391-7dac2534ecde" containerName="copy"
Sep 19 08:49:18.343990 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.343975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.346351 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.346331    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lnnkg\"/\"default-dockercfg-gczgz\""
Sep 19 08:49:18.347282 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.347258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lnnkg\"/\"openshift-service-ca.crt\""
Sep 19 08:49:18.347384 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.347285    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lnnkg\"/\"kube-root-ca.crt\""
Sep 19 08:49:18.351504 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.351483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"]
Sep 19 08:49:18.423827 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.423782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-proc\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.423998 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.423856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-podres\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.423998 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.423877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-lib-modules\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.423998 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.423906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-sys\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.423998 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.423931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbzd\" (UniqueName: \"kubernetes.io/projected/6726a15e-c527-41aa-ba37-455fe78999a2-kube-api-access-5zbzd\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524493 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-podres\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-lib-modules\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-podres\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-sys\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbzd\" (UniqueName: \"kubernetes.io/projected/6726a15e-c527-41aa-ba37-455fe78999a2-kube-api-access-5zbzd\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-proc\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524627 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-lib-modules\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524867 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-proc\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.524867 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.524743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6726a15e-c527-41aa-ba37-455fe78999a2-sys\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.532346 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.532324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbzd\" (UniqueName: \"kubernetes.io/projected/6726a15e-c527-41aa-ba37-455fe78999a2-kube-api-access-5zbzd\") pod \"perf-node-gather-daemonset-xscrf\" (UID: \"6726a15e-c527-41aa-ba37-455fe78999a2\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.654235 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.654174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:18.923208 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:18.923187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"]
Sep 19 08:49:18.925835 ip-10-0-140-141 kubenswrapper[2579]: W0919 08:49:18.925805    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6726a15e_c527_41aa_ba37_455fe78999a2.slice/crio-3f24a8f943077c712c1657679ad92fd93a329f5d11fe162ba785df5d43e8abc1 WatchSource:0}: Error finding container 3f24a8f943077c712c1657679ad92fd93a329f5d11fe162ba785df5d43e8abc1: Status 404 returned error can't find the container with id 3f24a8f943077c712c1657679ad92fd93a329f5d11fe162ba785df5d43e8abc1
Sep 19 08:49:19.271682 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.271588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-84qvd_5c35a6f7-ada6-4695-a9d8-6e3d8560790a/dns/0.log"
Sep 19 08:49:19.291859 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.291837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-84qvd_5c35a6f7-ada6-4695-a9d8-6e3d8560790a/kube-rbac-proxy/0.log"
Sep 19 08:49:19.339771 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.339748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf" event={"ID":"6726a15e-c527-41aa-ba37-455fe78999a2","Type":"ContainerStarted","Data":"69dac00e51e5c4f6ae76fbcd07f138014b8e59ef44e95a510e2ce9dbdb2a67fb"}
Sep 19 08:49:19.339873 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.339790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf" event={"ID":"6726a15e-c527-41aa-ba37-455fe78999a2","Type":"ContainerStarted","Data":"3f24a8f943077c712c1657679ad92fd93a329f5d11fe162ba785df5d43e8abc1"}
Sep 19 08:49:19.339912 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.339872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:19.355854 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.355812    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf" podStartSLOduration=1.355800559 podStartE2EDuration="1.355800559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:19.354999258 +0000 UTC m=+1710.020134802" watchObservedRunningTime="2026-09-19 08:49:19.355800559 +0000 UTC m=+1710.020936144"
Sep 19 08:49:19.381900 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.381882    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wwjml_add4b376-afa3-493c-aeb1-916bfeacbf89/dns-node-resolver/0.log"
Sep 19 08:49:19.929304 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:19.929278    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vmpk4_0a6f6ec9-e47c-4957-b28b-14014d1af467/node-ca/0.log"
Sep 19 08:49:20.614409 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:20.614385    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76cf86969b-jt5mj_8a190dfe-d0b5-426c-a225-ad3caed916f1/router/0.log"
Sep 19 08:49:20.976442 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:20.976372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9zgzq_991799e8-945e-4c15-aeaf-a664b5f10700/serve-healthcheck-canary/0.log"
Sep 19 08:49:21.379449 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:21.379426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvd6q_e58663f3-4077-4718-84d0-016a4127b3f5/kube-rbac-proxy/0.log"
Sep 19 08:49:21.398397 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:21.398376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvd6q_e58663f3-4077-4718-84d0-016a4127b3f5/exporter/0.log"
Sep 19 08:49:21.416839 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:21.416819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvd6q_e58663f3-4077-4718-84d0-016a4127b3f5/extractor/0.log"
Sep 19 08:49:25.352954 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:25.352922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-xscrf"
Sep 19 08:49:29.482194 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.482133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/kube-multus-additional-cni-plugins/0.log"
Sep 19 08:49:29.501066 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.501041    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/egress-router-binary-copy/0.log"
Sep 19 08:49:29.519567 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.519545    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/cni-plugins/0.log"
Sep 19 08:49:29.538188 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.538168    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/bond-cni-plugin/0.log"
Sep 19 08:49:29.557836 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.557820    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/routeoverride-cni/0.log"
Sep 19 08:49:29.576520 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.576503    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/whereabouts-cni-bincopy/0.log"
Sep 19 08:49:29.595557 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.595540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x6bcm_193907a3-f925-4deb-a7c5-4707d8ee6788/whereabouts-cni/0.log"
Sep 19 08:49:29.650543 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.650523    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fl2k7_e8a9c0f8-f161-40df-8ad5-7b6059dd1baa/kube-multus/0.log"
Sep 19 08:49:29.798815 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.798794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tz7f6_ff198034-e56f-44da-8427-9046ff196291/network-metrics-daemon/0.log"
Sep 19 08:49:29.817587 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:29.817564    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tz7f6_ff198034-e56f-44da-8427-9046ff196291/kube-rbac-proxy/0.log"
Sep 19 08:49:30.598189 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.598160    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-controller/0.log"
Sep 19 08:49:30.616316 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.616290    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/1.log"
Sep 19 08:49:30.616757 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.616654    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovn-acl-logging/0.log"
Sep 19 08:49:30.634134 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.634115    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/kube-rbac-proxy-node/0.log"
Sep 19 08:49:30.653095 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.653067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/kube-rbac-proxy-ovn-metrics/0.log"
Sep 19 08:49:30.673404 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.673389    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/northd/0.log"
Sep 19 08:49:30.694603 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.694585    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/nbdb/0.log"
Sep 19 08:49:30.715001 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.714984    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/sbdb/0.log"
Sep 19 08:49:30.748007 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:30.747992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bkpds_202713c0-1ac1-483e-8389-57e9d34c23e4/ovnkube-controller/0.log"
Sep 19 08:49:32.059945 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:32.059917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-8mrzh_80301f23-031b-488e-991d-efc52d87ccc6/check-endpoints/0.log"
Sep 19 08:49:32.126385 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:32.126363    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tqgz5_eadec7b8-90a1-465f-8731-eed5d4bdfd49/network-check-target-container/0.log"
Sep 19 08:49:32.983352 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:32.983324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qc9bm_a13f1d56-3edb-463f-bee7-5a0b2af6423b/iptables-alerter/0.log"
Sep 19 08:49:33.594316 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:33.594289    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-d7k5h_a314ace4-fd72-46a7-96e0-81e1bb51a464/tuned/0.log"
Sep 19 08:49:35.302804 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:35.302779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-85c5b_36193d7a-8e53-4f3a-8214-876bb5d56eec/cluster-samples-operator/0.log"
Sep 19 08:49:35.319854 ip-10-0-140-141 kubenswrapper[2579]: I0919 08:49:35.319831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-85c5b_36193d7a-8e53-4f3a-8214-876bb5d56eec/cluster-samples-operator-watch/0.log"