Sep 17 17:56:51.461979 ip-10-0-143-209 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:56:55.362812 ip-10-0-143-209 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:55.362812 ip-10-0-143-209 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:56:55.362812 ip-10-0-143-209 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:55.362812 ip-10-0-143-209 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:56:55.362812 ip-10-0-143-209 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:55.363673 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.362892    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:56:55.367343 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367324    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:55.367343 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367343    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367347    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367350    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367353    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367356    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367359    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367362    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367366    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367370    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367379    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367382    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367385    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367388    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367392    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367396    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367399    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367402    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367405    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367408    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:55.367417 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367411    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367414    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367417    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367420    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367423    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367426    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367430    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367433    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367435    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367438    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367441    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367444    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367447    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367451    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367453    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367456    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367459    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367462    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367465    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367468    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:55.367913 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367470    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367473    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367476    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367478    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367481    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367484    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367486    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367491    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367495    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367499    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367501    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367504    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367507    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367509    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367513    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367516    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367519    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367521    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367524    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:55.368465 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367527    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367530    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367532    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367535    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367538    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367540    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367543    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367545    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367548    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367551    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367554    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367557    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367560    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367563    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367565    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367568    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367571    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367573    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367576    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:55.369255 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367579    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367583    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367586    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367589    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367592    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367594    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367597    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367600    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367604    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367606    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367609    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367612    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367615    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367618    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367620    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367623    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367626    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367628    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367631    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367634    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:55.369817 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.367652    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369881    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369896    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369900    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369904    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369908    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369911    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369915    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369918    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369921    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369924    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369927    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369930    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369933    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369936    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369939    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369942    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369944    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369947    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369950    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:55.370352 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369953    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369955    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369958    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369961    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369964    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369967    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369970    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369972    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369975    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369978    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369982    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369987    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369991    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369994    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.369999    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370002    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370005    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370008    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370011    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370013    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:55.370906 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370016    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370019    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370022    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370025    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370027    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370030    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370033    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370035    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370038    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370041    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370044    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370046    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370049    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370052    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370055    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370059    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370062    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370065    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370068    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:55.371454 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370071    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370075    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370079    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370082    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370084    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370087    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370091    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370095    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370098    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370101    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370104    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370107    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370109    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370112    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370115    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370118    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370120    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370123    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370125    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370129    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:55.371970 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370131    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370134    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370137    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370140    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370143    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370148    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370151    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370154    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370158    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370161    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370164    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370167    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370170    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370173    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370175    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370178    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370181    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370184    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370186    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:55.372490 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370190    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370193    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370627    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370633    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370651    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370654    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370657    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370660    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370663    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370666    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370669    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370672    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370675    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370678    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370680    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370683    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370686    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370689    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370692    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:55.373039 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370697    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370701    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370705    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370708    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370711    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370714    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370717    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370720    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370723    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370726    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370729    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370732    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370734    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370738    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370741    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370744    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370746    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370749    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370752    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:55.373594 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370755    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370758    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370761    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370763    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370766    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370769    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370771    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370774    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370777    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370779    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370782    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370785    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370788    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370795    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370798    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370801    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370804    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370806    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370809    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370812    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:55.374122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370815    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370818    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370821    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370824    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370833    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370837    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370841    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370844    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370847    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370849    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370852    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370856    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370859    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370862    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370866    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370869    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370872    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370875    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370878    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370880    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:55.374716 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370883    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370886    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370889    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370892    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370895    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370898    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370902    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370905    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370908    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370911    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370913    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370917    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370921    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370924    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370927    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370930    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370933    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370935    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370938    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:55.375231 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370941    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370944    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370947    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.370950    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371039    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371091    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371099    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371104    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371109    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371113    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371117    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371122    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371125    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371129    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371133    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371136    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371144    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371147    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371150    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371153    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371158    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371162    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371166    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371169    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:56:55.375744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371172    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371176    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371180    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371184    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371188    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371191    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371194    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371198    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371201    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371204    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371208    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371211    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371215    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371219    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371222    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371225    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371228    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371231    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371236    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371239    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371242    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371245    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371248    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371252    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:56:55.376354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371257    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371260    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371263    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371267    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371269    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371273    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371276    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371280    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371283    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371286    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371290    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371293    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371296    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371299    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371303    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371306    2577 flags.go:64] FLAG: --help="false"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371309    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371313    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371316    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371319    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371323    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371327    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371330    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:56:55.376986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371333    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371336    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371339    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371342    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371345    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371348    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371351    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371354    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371358    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371362    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371365    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371368    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371372    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371375    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371380    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371384    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371389    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371392    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371395    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371399    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371402    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371406    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371410    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371413    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371418    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:56:55.377617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371421    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371424    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371427    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371431    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371434    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371437    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371440    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371448    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371452    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371455    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371459    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371463    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371469    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371472    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371475    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371478    2577 flags.go:64] FLAG: --port="10250"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371483    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371486    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b5ccd8d2b770364a"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371489    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371493    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371496    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371499    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371503    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371507    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:56:55.378329 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371512    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371515    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371519    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371522    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371525    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371528    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371531    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371534    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371538    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371541    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371544    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371547    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371550    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371553    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371557    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371560    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371563    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371566    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371569    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371572    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371575    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371578    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371584    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371587    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371592    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:56:55.379059 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371595    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371598    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371601    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371604    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371608    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371611    2577 flags.go:64] FLAG: --v="2"
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371615    2577 flags.go:64] FLAG: --version="false"
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371621    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371626    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.371629    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371772    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371776    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371780    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371783    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371786    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371789    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371793    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371797    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371801    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371804    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371808    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:55.379718 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371811    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371814    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371817    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371820    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371823    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371826    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371829    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371831    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371834    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371837    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371842    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371845    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371847    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371850    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371853    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371856    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371859    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371862    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371866    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371870    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:55.380286 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371873    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371875    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371878    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371881    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371884    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371887    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371890    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371893    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371896    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371899    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371902    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371905    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371908    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371910    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371913    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371916    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371919    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371922    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371925    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371927    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:55.380835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371931    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371933    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371938    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371941    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371944    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371946    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371949    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371952    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371956    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371960    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371964    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371968    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371971    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371975    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371978    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371981    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371984    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371987    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371990    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:55.381373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371993    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371996    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.371999    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372002    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372006    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372010    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372013    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372016    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372019    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372022    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372025    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372028    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372032    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372035    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372038    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372042    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372045    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372048    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372051    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372053    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:55.381935 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372056    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372059    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372062    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372066    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372069    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372071    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372075    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372078    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.372081    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.373165    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.380551    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.380568    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.380594    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.380604    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380682    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:55.382483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380687    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380691    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380695    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380698    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380701    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380704    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380707    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380710    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380713    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380716    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380719    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380722    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380725    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380729    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380734    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380737    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380741    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380744    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380747    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380750    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:55.382899 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380754    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380758    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380762    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380766    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380769    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380772    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380776    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380779    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380783    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380786    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380789    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380792    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380795    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380798    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380801    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380804    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380806    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380810    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380812    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:55.383418 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380815    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380818    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380821    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380824    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380828    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380831    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380833    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380836    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380839    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380843    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380846    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380850    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380853    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380856    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380859    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380862    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380865    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380869    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380871    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:55.383955 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380875    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380878    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380881    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380886    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380890    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380893    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380896    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380899    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380902    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380905    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380908    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380911    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380914    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380917    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380919    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380922    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380926    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380929    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380932    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:55.384448 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380935    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380938    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380941    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380944    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380947    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380950    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380953    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380955    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380958    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380961    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380964    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380967    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380970    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380974    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380977    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380980    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380984    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380987    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380990    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380992    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:55.384968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.380995    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.381000    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381164    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381169    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381173    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381176    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381179    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381183    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381186    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381189    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381192    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381195    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381198    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381201    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:55.385485 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381204    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381207    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381210    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381213    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381216    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381219    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381222    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381224    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381227    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381230    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381233    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381237    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381240    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381244    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381247    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381250    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381253    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381255    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381258    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381261    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:55.385883 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381264    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381267    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381270    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381273    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381275    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381278    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381281    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381283    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381286    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381289    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381292    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381295    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381298    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381300    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381303    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381306    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381308    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381311    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381314    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381317    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:55.386403 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381320    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381323    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381325    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381328    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381332    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381335    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381338    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381341    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381344    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381347    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381349    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381354    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381358    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381361    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381364    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381367    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381370    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381374    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381378    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381381    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:55.386952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381384    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381387    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381390    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381393    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381396    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381399    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381402    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381404    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381407    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381410    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381413    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381416    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381420    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381424    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381427    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381430    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381433    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381437    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381439    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:55.387478 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381442    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381445    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381448    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381451    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381454    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381456    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381459    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:55.381462    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.381467    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.381573    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.383861    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.384977    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.385081    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:55.388122 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.385115    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:55.388530 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.388198    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:55.390133 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.390109    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:55.401176 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.401152    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:56:55.402800 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.402781    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 17:56:55.403710 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.403691    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:56:55.406216 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.406195    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b15d03e8-7d40-4aef-9de8-a0908d1cf810:/dev/nvme0n1p4 cfbb2976-edbf-4742-9b39-85f6bc97c589:/dev/nvme0n1p3]
Sep 17 17:56:55.406298 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.406215    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:56:55.411683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.411506    2577 manager.go:216] Machine: {Timestamp:2026-09-17 17:56:55.410604067 +0000 UTC m=+2.253767210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101204 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258893f9b1e83a8cb487821711ce2c SystemUUID:ec258893-f9b1-e83a-8cb4-87821711ce2c BootID:91961bf8-b7bd-4762-9260-d72263352e15 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:5d:e8:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:5d:e8:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:1a:e1:24:cb:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:56:55.411772 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.411684    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:56:55.411805 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.411789    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:56:55.413369 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.413339    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:56:55.413546 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.413373    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:56:55.414880 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.414866    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:56:55.414939 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.414888    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:56:55.414939 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.414900    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:55.414939 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.414916    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:55.416740 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.416727    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:56:55.419884 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.419870    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:55.420063 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420050    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:56:55.420132 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420098    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:56:55.420132 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420115    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:56:55.420132 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420131    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:56:55.420263 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420145    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:56:55.420263 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420159    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:56:55.420597 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420583    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:55.420689 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.420611    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:55.423041 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.423015    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:56:55.425462 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.425448    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:56:55.425506 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.425481    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:56:55.426466 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426455    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:56:55.426502 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426472    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:56:55.426502 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426479    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:56:55.426502 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426484    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:56:55.426502 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426494    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:56:55.426502 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426503    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426524    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426530    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426537    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426543    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426552    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426557    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426583    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:56:55.426651 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426592    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:56:55.426981 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.426963    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:55.432013 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.431998    2577 server.go:1268] "Started kubelet"
Sep 17 17:56:55.432161 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.432098    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:56:55.432260 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.432160    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:56:55.432260 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.432218    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:56:55.466297 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.466232    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:56:55.466588 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.466526    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:56:55.466812 ip-10-0-143-209 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:56:55.467698 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.466868    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:56:55.468512 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.468490    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:56:55.468665 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.468642    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:56:55.470700 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.470677    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:56:55.473803 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.472785    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6564c5a4ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.431972079 +0000 UTC m=+2.275135219,LastTimestamp:2026-09-17 17:56:55.431972079 +0000 UTC m=+2.275135219,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.475103 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475087    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:56:55.475103 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475099    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:55.475505 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.475288    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:55.475505 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475715    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:56:55.475505 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475729    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:56:55.475505 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475744    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:56:55.476007 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475908    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:56:55.476007 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.475934    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:56:55.476272 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476254    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:56:55.476272 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476273    2577 factory.go:55] Registering systemd factory
Sep 17 17:56:55.476393 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476283    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:56:55.476555 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476539    2577 factory.go:153] Registering CRI-O factory
Sep 17 17:56:55.476611 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476561    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:56:55.476611 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476596    2577 factory.go:103] Registering Raw factory
Sep 17 17:56:55.476611 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.476611    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:56:55.477444 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.477357    2577 manager.go:349] Starting recovery of all containers
Sep 17 17:56:55.485472 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.485445    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:56:55.485590 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.485520    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:56:55.489315 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.489137    2577 manager.go:354] Recovery completed
Sep 17 17:56:55.494546 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.494533    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.498267 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.498242    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.498332 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.498287    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.498332 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.498306    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.500714 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.500698    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:56:55.500790 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.500717    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:56:55.500790 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.500736    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:55.500790 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.500704    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.500995 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.500906    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6568b93162  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.49826493 +0000 UTC m=+2.341428077,LastTimestamp:2026-09-17 17:56:55.49826493 +0000 UTC m=+2.341428077,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.501896 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.501878    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.501953 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.501925    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.501953 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.501939    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.502168 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.502155    2577 policy_none.go:49] "None policy: Start"
Sep 17 17:56:55.502199 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.502174    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:56:55.502199 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.502191    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:56:55.503385 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.503367    2577 policy_none.go:47] "Start"
Sep 17 17:56:55.512969 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.512902    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6568b9a87c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-209.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.49829542 +0000 UTC m=+2.341458563,LastTimestamp:2026-09-17 17:56:55.49829542 +0000 UTC m=+2.341458563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.515118 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.515024    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6568b9ebfb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-209.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.498312699 +0000 UTC m=+2.341475842,LastTimestamp:2026-09-17 17:56:55.498312699 +0000 UTC m=+2.341475842,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.524310 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.524218    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-209.ec2.internal.18d62d6568b93162\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6568b93162  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.49826493 +0000 UTC m=+2.341428077,LastTimestamp:2026-09-17 17:56:55.501895916 +0000 UTC m=+2.345059058,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.531618    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-209.ec2.internal.18d62d6568b9a87c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6568b9a87c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-209.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.49829542 +0000 UTC m=+2.341458563,LastTimestamp:2026-09-17 17:56:55.501933551 +0000 UTC m=+2.345096693,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.539047    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-209.ec2.internal.18d62d6568b9ebfb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d6568b9ebfb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-209.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.498312699 +0000 UTC m=+2.341475842,LastTimestamp:2026-09-17 17:56:55.501943634 +0000 UTC m=+2.345106775,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540236    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.540281    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540295    2577 server.go:87] "Starting device plugin registration server"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540555    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540581    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540729    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540862    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.540871    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.541445    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:56:55.544971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.541500    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:55.546764 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.546699    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-209.ec2.internal.18d62d656b5cd9b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-209.ec2.internal,UID:ip-10-0-143-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-209.ec2.internal,},FirstTimestamp:2026-09-17 17:56:55.542544825 +0000 UTC m=+2.385707956,LastTimestamp:2026-09-17 17:56:55.542544825 +0000 UTC m=+2.385707956,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-209.ec2.internal,}"
Sep 17 17:56:55.577350 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.577317    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:56:55.578638 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.578620    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:56:55.578714 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.578687    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:56:55.578714 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.578713    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:56:55.578789 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.578762    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:56:55.581215 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.581191    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 17:56:55.581984 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.581966    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhxxr"
Sep 17 17:56:55.594099 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.594081    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhxxr"
Sep 17 17:56:55.641473 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.641412    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.642472 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.642453    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.642558 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.642495    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.642558 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.642508    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.642558 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.642532    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.679346 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.679300    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"]
Sep 17 17:56:55.679535 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.679401    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.681252 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.681234    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.681340 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.681268    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.681340 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.681279    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.682656 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.682642    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.682822 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.682807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.682863 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.682848    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.683449 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.683431    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.683525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.683464    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.683525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.683474    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.683525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.683496    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.683525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.683480    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.683525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.683511    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.684862 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.684846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.684957 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.684876    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:55.685665 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.685648    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:55.685752 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.685679    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:55.685752 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.685698    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:55.690093 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.690075    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.690184 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.690103    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-209.ec2.internal\": node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:55.713481 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.713459    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-209.ec2.internal\" not found" node="ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.718272 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.718256    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-209.ec2.internal\" not found" node="ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.734551 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.734530    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:55.778200 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.778173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc4ad02cbeb2d0216949bfc307678343-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal\" (UID: \"bc4ad02cbeb2d0216949bfc307678343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.778200 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.778201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799c76a8366626239baf308b23899d1e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-209.ec2.internal\" (UID: \"799c76a8366626239baf308b23899d1e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.778352 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.778218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc4ad02cbeb2d0216949bfc307678343-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal\" (UID: \"bc4ad02cbeb2d0216949bfc307678343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.834833 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.834801    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:55.879015 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.878994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799c76a8366626239baf308b23899d1e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-209.ec2.internal\" (UID: \"799c76a8366626239baf308b23899d1e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.879151 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.879022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc4ad02cbeb2d0216949bfc307678343-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal\" (UID: \"bc4ad02cbeb2d0216949bfc307678343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.879151 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.879040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc4ad02cbeb2d0216949bfc307678343-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal\" (UID: \"bc4ad02cbeb2d0216949bfc307678343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.879151 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.879082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc4ad02cbeb2d0216949bfc307678343-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal\" (UID: \"bc4ad02cbeb2d0216949bfc307678343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.879151 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.879126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc4ad02cbeb2d0216949bfc307678343-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal\" (UID: \"bc4ad02cbeb2d0216949bfc307678343\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.879303 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:55.879147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799c76a8366626239baf308b23899d1e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-209.ec2.internal\" (UID: \"799c76a8366626239baf308b23899d1e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:55.935082 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:55.935015    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:56.017043 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.017013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:56.021947 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.021810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:56.035694 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.035673    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:56.139441 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.139393    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:56.239807 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.239741    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:56.280807 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.280778    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:56.340974 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.340947    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-209.ec2.internal\" not found"
Sep 17 17:56:56.343375 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.343355    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:56.375347 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.375319    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:56.384184 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.384166    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:56:56.384298 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.384275    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:56:56.384365 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.384285    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:56:56.384407 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.384341    2577 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a9fa84e90973c412899cf83e0fc51972-6afd5615cbb7efcf.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.143.209:45748->67.202.13.31:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:56.384407 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.384382    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal"
Sep 17 17:56:56.409241 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.409220    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:56.420388 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.420371    2577 apiserver.go:52] "Watching apiserver"
Sep 17 17:56:56.428398 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.428376    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:56:56.428763 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.428744    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-hd5cc","openshift-multus/network-metrics-daemon-9flf9","openshift-network-operator/iptables-alerter-dcqg2","openshift-ovn-kubernetes/ovnkube-node-mlxrv","kube-system/global-pull-secret-syncer-j4qs8","kube-system/konnectivity-agent-bgkwf","kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wt26z","openshift-dns/node-resolver-k6fsk","openshift-image-registry/node-ca-4th94","openshift-multus/multus-additional-cni-plugins-29btz","openshift-network-diagnostics/network-check-target-b9jfv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal"]
Sep 17 17:56:56.430464 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.430447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.431628 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.431611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.433002 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.432985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.433055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.433035    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:56.434201 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.434185    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.434304 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.434289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.434758 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.434735    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.434851 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.434801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:56:56.435658 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.435636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.435736 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.435721    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:56:56.436300 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.436286    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:56:56.436625 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.436612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-642zt\""
Sep 17 17:56:56.436795 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.436779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.437822 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.437805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.437979 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.437957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9ddlj\""
Sep 17 17:56:56.438075 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438004    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:56:56.438075 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.438192 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.438192 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p88j8\""
Sep 17 17:56:56.438739 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438721    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.438819 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.438819 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.438808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:56:56.439539 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.439524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.440011 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.439989    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r25n\""
Sep 17 17:56:56.440101 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:56:56.440212 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440192    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:56:56.440409 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.440496 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440474    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.440549 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:56:56.440549 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:56:56.440682 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440541    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:56:56.440947 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.440932    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bfgxz\""
Sep 17 17:56:56.441153 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.441139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:56.441211 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.441189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:56:56.445603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.444784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.445603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.444812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8cf8f\""
Sep 17 17:56:56.445603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.445122    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.445603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.445308    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ddrtm\""
Sep 17 17:56:56.445603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.445434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.445603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.445527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.446367 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.446155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.448720 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.448699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.450224 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.450202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:56:56.450224 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.450213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:56:56.450392 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.450274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:56:56.450392 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.450326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:56:56.450392 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.450339    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v9dhj\""
Sep 17 17:56:56.450392 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.450326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:56:56.450392 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.450331    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:56:56.451310 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.451288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:56:56.451434 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.451390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbc9q\""
Sep 17 17:56:56.451767 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.451747    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:56:56.475662 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.475632    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:56.476779 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.476759    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:56:56.483261 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-device-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.483387 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-run-netns\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483387 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-log-socket\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483387 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbkv\" (UniqueName: \"kubernetes.io/projected/f3c3c5be-9d14-4960-a097-2a89ebaae22a-kube-api-access-qxbkv\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.483520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5077d071-443f-4f95-b556-2ac43e51d310-konnectivity-ca\") pod \"konnectivity-agent-bgkwf\" (UID: \"5077d071-443f-4f95-b556-2ac43e51d310\") " pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.483520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-sys\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.483520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-daemon-config\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.483520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5077d071-443f-4f95-b556-2ac43e51d310-agent-certs\") pod \"konnectivity-agent-bgkwf\" (UID: \"5077d071-443f-4f95-b556-2ac43e51d310\") " pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-node-log\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-socket-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-registration-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-ovnkube-config\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-cnibin\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-etc-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysctl-d\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.483774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysctl-conf\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-kubernetes\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-systemd\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-kubelet\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-systemd-units\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c047389-08f6-46f9-860e-47ca8dade973-tmp-dir\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-host\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-hostroot\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.483980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-multus-certs\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nrz\" (UniqueName: \"kubernetes.io/projected/1c047389-08f6-46f9-860e-47ca8dade973-kube-api-access-46nrz\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f52e62dd-57e7-4a52-a6d4-bfe36454a697-tmp\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b8b260d-874d-4242-88a5-7f3aa34a4126-cni-binary-copy\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7h2z\" (UniqueName: \"kubernetes.io/projected/3b8b260d-874d-4242-88a5-7f3aa34a4126-kube-api-access-m7h2z\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-ovn\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.484250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cni-binary-copy\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-tuned\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrq2\" (UniqueName: \"kubernetes.io/projected/761dc4fd-5005-48a1-9874-bfa11455f308-kube-api-access-pwrq2\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglrn\" (UniqueName: \"kubernetes.io/projected/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-kube-api-access-xglrn\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-cni-netd\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-cni-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-conf-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g692m\" (UniqueName: \"kubernetes.io/projected/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-kube-api-access-g692m\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysconfig\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-serviceca\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-kubelet\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-ovnkube-script-lib\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cmh\" (UniqueName: \"kubernetes.io/projected/1e73f216-d440-41b0-a209-65eefbe31903-kube-api-access-p8cmh\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-system-cni-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-socket-dir-parent\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.485131 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-system-cni-dir\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.484744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-tuning-conf-dir\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxh9b\" (UniqueName: \"kubernetes.io/projected/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-kube-api-access-dxh9b\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-iptables-alerter-script\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-env-overrides\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-k8s-cni-cncf-io\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-etc-kubernetes\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-sys-fs\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.485603 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-var-lib-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-host\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-host-slash\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-slash\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-systemd\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e73f216-d440-41b0-a209-65eefbe31903-ovn-node-metrics-cert\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cnibin\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-run\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-var-lib-kubelet\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-cni-bin\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c047389-08f6-46f9-860e-47ca8dade973-hosts-file\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-modprobe-d\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4m7\" (UniqueName: \"kubernetes.io/projected/f52e62dd-57e7-4a52-a6d4-bfe36454a697-kube-api-access-zb4m7\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.485919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-cni-multus\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.486468 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4acb9960-a709-42b3-9f79-ef953d15e558-kubelet-config\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.486468 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4acb9960-a709-42b3-9f79-ef953d15e558-dbus\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.486468 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.485990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-netns\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.486468 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.486004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-cni-bin\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.486468 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.486030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-lib-modules\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.502200 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.502178    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:56.586365 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f52e62dd-57e7-4a52-a6d4-bfe36454a697-tmp\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b8b260d-874d-4242-88a5-7f3aa34a4126-cni-binary-copy\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7h2z\" (UniqueName: \"kubernetes.io/projected/3b8b260d-874d-4242-88a5-7f3aa34a4126-kube-api-access-m7h2z\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-ovn\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cni-binary-copy\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-tuned\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrq2\" (UniqueName: \"kubernetes.io/projected/761dc4fd-5005-48a1-9874-bfa11455f308-kube-api-access-pwrq2\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:56.586583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglrn\" (UniqueName: \"kubernetes.io/projected/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-kube-api-access-xglrn\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-cni-netd\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-cni-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586716    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-ovn\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-cni-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.587021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.586939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b8b260d-874d-4242-88a5-7f3aa34a4126-cni-binary-copy\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.587389 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-cni-netd\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.587389 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-conf-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.587389 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g692m\" (UniqueName: \"kubernetes.io/projected/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-kube-api-access-g692m\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.587389 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-conf-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.587389 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysconfig\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.587389 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-serviceca\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.587650 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-kubelet\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.587650 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-ovnkube-script-lib\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.587807 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-kubelet\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.587984 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.587966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cmh\" (UniqueName: \"kubernetes.io/projected/1e73f216-d440-41b0-a209-65eefbe31903-kube-api-access-p8cmh\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.588041 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-system-cni-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.588087 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-socket-dir-parent\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.588087 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-system-cni-dir\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.588177 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-tuning-conf-dir\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.588411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-tuning-conf-dir\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.588411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysconfig\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.588411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cni-binary-copy\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.588411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-system-cni-dir\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.588411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-system-cni-dir\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.588411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-socket-dir-parent\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.588750 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxh9b\" (UniqueName: \"kubernetes.io/projected/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-kube-api-access-dxh9b\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.588750 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-iptables-alerter-script\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.588750 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-env-overrides\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.588968 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.588935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-ovnkube-script-lib\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-env-overrides\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-serviceca\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-k8s-cni-cncf-io\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-etc-kubernetes\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-sys-fs\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-var-lib-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-etc-kubernetes\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-k8s-cni-cncf-io\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-host\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-sys-fs\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-host-slash\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.589545    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-slash\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:56:56.589683 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.589663    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:57.089623829 +0000 UTC m=+3.932786975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-host\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-systemd\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-var-lib-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e73f216-d440-41b0-a209-65eefbe31903-ovn-node-metrics-cert\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-slash\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cnibin\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-systemd\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-run\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-var-lib-kubelet\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-cni-bin\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cnibin\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c047389-08f6-46f9-860e-47ca8dade973-hosts-file\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-run-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.589552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-host-slash\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-modprobe-d\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.590513 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4m7\" (UniqueName: \"kubernetes.io/projected/f52e62dd-57e7-4a52-a6d4-bfe36454a697-kube-api-access-zb4m7\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-cni-multus\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4acb9960-a709-42b3-9f79-ef953d15e558-kubelet-config\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-run\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4acb9960-a709-42b3-9f79-ef953d15e558-dbus\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-netns\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-cni-bin\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-var-lib-kubelet\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-lib-modules\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-device-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-cni-bin\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-run-netns\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-tuned\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-run-netns\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f52e62dd-57e7-4a52-a6d4-bfe36454a697-tmp\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4acb9960-a709-42b3-9f79-ef953d15e558-dbus\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-log-socket\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbkv\" (UniqueName: \"kubernetes.io/projected/f3c3c5be-9d14-4960-a097-2a89ebaae22a-kube-api-access-qxbkv\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.591391 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5077d071-443f-4f95-b556-2ac43e51d310-konnectivity-ca\") pod \"konnectivity-agent-bgkwf\" (UID: \"5077d071-443f-4f95-b556-2ac43e51d310\") " pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-sys\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-netns\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-cni-multus\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-log-socket\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-modprobe-d\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4acb9960-a709-42b3-9f79-ef953d15e558-kubelet-config\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-daemon-config\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5077d071-443f-4f95-b556-2ac43e51d310-agent-certs\") pod \"konnectivity-agent-bgkwf\" (UID: \"5077d071-443f-4f95-b556-2ac43e51d310\") " pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.590991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-node-log\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-socket-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-registration-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-ovnkube-config\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.592240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-cnibin\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-cnibin\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1e73f216-d440-41b0-a209-65eefbe31903-ovnkube-config\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-etc-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysctl-d\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysctl-conf\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-kubernetes\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysctl-conf\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5077d071-443f-4f95-b556-2ac43e51d310-konnectivity-ca\") pod \"konnectivity-agent-bgkwf\" (UID: \"5077d071-443f-4f95-b556-2ac43e51d310\") " pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.591992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-systemd\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b8b260d-874d-4242-88a5-7f3aa34a4126-multus-daemon-config\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-systemd\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-kubelet\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-kubelet\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-kubernetes\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-systemd-units\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.593121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c047389-08f6-46f9-860e-47ca8dade973-tmp-dir\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-host\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-node-log\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-hostroot\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-multus-certs\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nrz\" (UniqueName: \"kubernetes.io/projected/1c047389-08f6-46f9-860e-47ca8dade973-kube-api-access-46nrz\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-socket-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.593061    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-host\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e73f216-d440-41b0-a209-65eefbe31903-ovn-node-metrics-cert\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.593140    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:57.093097494 +0000 UTC m=+3.936260620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.592458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-var-lib-cni-bin\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-lib-modules\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-device-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-etc-sysctl-d\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.594023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52e62dd-57e7-4a52-a6d4-bfe36454a697-sys\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c047389-08f6-46f9-860e-47ca8dade973-hosts-file\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-etc-openvswitch\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-host-run-multus-certs\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b8b260d-874d-4242-88a5-7f3aa34a4126-hostroot\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3c3c5be-9d14-4960-a097-2a89ebaae22a-registration-dir\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593437    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xhq6t"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1e73f216-d440-41b0-a209-65eefbe31903-systemd-units\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c047389-08f6-46f9-860e-47ca8dade973-tmp-dir\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.593810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-iptables-alerter-script\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.594759 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.594671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5077d071-443f-4f95-b556-2ac43e51d310-agent-certs\") pod \"konnectivity-agent-bgkwf\" (UID: \"5077d071-443f-4f95-b556-2ac43e51d310\") " pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.599167 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.599129    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:51:55 +0000 UTC" deadline="2028-06-08 12:58:57.884826864 +0000 UTC"
Sep 17 17:56:56.599167 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.599165    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15115h2m1.285664269s"
Sep 17 17:56:56.601861 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.601843    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:56.601959 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.601866    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:56.601959 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.601879    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:56.602073 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:56.601975    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:57.101958963 +0000 UTC m=+3.945122106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:56.603894 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.603872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cmh\" (UniqueName: \"kubernetes.io/projected/1e73f216-d440-41b0-a209-65eefbe31903-kube-api-access-p8cmh\") pod \"ovnkube-node-mlxrv\" (UID: \"1e73f216-d440-41b0-a209-65eefbe31903\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.604358 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.604340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4m7\" (UniqueName: \"kubernetes.io/projected/f52e62dd-57e7-4a52-a6d4-bfe36454a697-kube-api-access-zb4m7\") pod \"tuned-wt26z\" (UID: \"f52e62dd-57e7-4a52-a6d4-bfe36454a697\") " pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.604684 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.604666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbkv\" (UniqueName: \"kubernetes.io/projected/f3c3c5be-9d14-4960-a097-2a89ebaae22a-kube-api-access-qxbkv\") pod \"aws-ebs-csi-driver-node-9z8xm\" (UID: \"f3c3c5be-9d14-4960-a097-2a89ebaae22a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.606109 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.606083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g692m\" (UniqueName: \"kubernetes.io/projected/977fa5b8-3f32-4f08-98f1-e8de6c3b4927-kube-api-access-g692m\") pod \"multus-additional-cni-plugins-29btz\" (UID: \"977fa5b8-3f32-4f08-98f1-e8de6c3b4927\") " pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.606484 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.606460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxh9b\" (UniqueName: \"kubernetes.io/projected/4e097696-45f7-4f39-a75b-0d5b5b51eb7c-kube-api-access-dxh9b\") pod \"node-ca-4th94\" (UID: \"4e097696-45f7-4f39-a75b-0d5b5b51eb7c\") " pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.607996 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.607973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrq2\" (UniqueName: \"kubernetes.io/projected/761dc4fd-5005-48a1-9874-bfa11455f308-kube-api-access-pwrq2\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:56.608080 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.607974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nrz\" (UniqueName: \"kubernetes.io/projected/1c047389-08f6-46f9-860e-47ca8dade973-kube-api-access-46nrz\") pod \"node-resolver-k6fsk\" (UID: \"1c047389-08f6-46f9-860e-47ca8dade973\") " pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.608290 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.608275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7h2z\" (UniqueName: \"kubernetes.io/projected/3b8b260d-874d-4242-88a5-7f3aa34a4126-kube-api-access-m7h2z\") pod \"multus-hd5cc\" (UID: \"3b8b260d-874d-4242-88a5-7f3aa34a4126\") " pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.609152 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.609133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglrn\" (UniqueName: \"kubernetes.io/projected/3f0b7652-f879-4ac8-a2a6-a7cf4042c0df-kube-api-access-xglrn\") pod \"iptables-alerter-dcqg2\" (UID: \"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df\") " pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.621667 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.621650    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xhq6t"
Sep 17 17:56:56.653169 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.653145    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799c76a8366626239baf308b23899d1e.slice/crio-fa1587b292abb1c62654eeaac8dfe17abfe80b386c97f8f51db417b434e2af98 WatchSource:0}: Error finding container fa1587b292abb1c62654eeaac8dfe17abfe80b386c97f8f51db417b434e2af98: Status 404 returned error can't find the container with id fa1587b292abb1c62654eeaac8dfe17abfe80b386c97f8f51db417b434e2af98
Sep 17 17:56:56.653582 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.653546    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4ad02cbeb2d0216949bfc307678343.slice/crio-8855f378cb325677d83e254ed702f3fd6ddfee059cb2d0aaff21b2506e74d58f WatchSource:0}: Error finding container 8855f378cb325677d83e254ed702f3fd6ddfee059cb2d0aaff21b2506e74d58f: Status 404 returned error can't find the container with id 8855f378cb325677d83e254ed702f3fd6ddfee059cb2d0aaff21b2506e74d58f
Sep 17 17:56:56.659999 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.659983    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:56:56.751907 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.751833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4th94"
Sep 17 17:56:56.758808 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.758786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e097696_45f7_4f39_a75b_0d5b5b51eb7c.slice/crio-65087530700d491ece3fe47c94e9bb0c5c43506fffb3cdd2176a655c441cbe40 WatchSource:0}: Error finding container 65087530700d491ece3fe47c94e9bb0c5c43506fffb3cdd2176a655c441cbe40: Status 404 returned error can't find the container with id 65087530700d491ece3fe47c94e9bb0c5c43506fffb3cdd2176a655c441cbe40
Sep 17 17:56:56.770168 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.770148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm"
Sep 17 17:56:56.776835 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.776816    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c3c5be_9d14_4960_a097_2a89ebaae22a.slice/crio-a575b37570b602fafe3a9b51157570a3de1d0af4070e78d18bd9cf2d69b3f56d WatchSource:0}: Error finding container a575b37570b602fafe3a9b51157570a3de1d0af4070e78d18bd9cf2d69b3f56d: Status 404 returned error can't find the container with id a575b37570b602fafe3a9b51157570a3de1d0af4070e78d18bd9cf2d69b3f56d
Sep 17 17:56:56.791363 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.791344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dcqg2"
Sep 17 17:56:56.798004 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.797979    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0b7652_f879_4ac8_a2a6_a7cf4042c0df.slice/crio-1e2a2df5219fa1ab613274e5829c86e815c38be66756aa7938f742d5eb68b8bf WatchSource:0}: Error finding container 1e2a2df5219fa1ab613274e5829c86e815c38be66756aa7938f742d5eb68b8bf: Status 404 returned error can't find the container with id 1e2a2df5219fa1ab613274e5829c86e815c38be66756aa7938f742d5eb68b8bf
Sep 17 17:56:56.809145 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.809129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:56:56.816483 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.816462    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e73f216_d440_41b0_a209_65eefbe31903.slice/crio-48945beee7f9ff4632d829999a76f28a70eba0c87aee2d45192179961b205977 WatchSource:0}: Error finding container 48945beee7f9ff4632d829999a76f28a70eba0c87aee2d45192179961b205977: Status 404 returned error can't find the container with id 48945beee7f9ff4632d829999a76f28a70eba0c87aee2d45192179961b205977
Sep 17 17:56:56.825159 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.825144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:56:56.831843 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.831823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5077d071_443f_4f95_b556_2ac43e51d310.slice/crio-e20c85a5e2e0affc6160bd89399ce90ec137fa20105ec8a8688cfc4a6ce13a4a WatchSource:0}: Error finding container e20c85a5e2e0affc6160bd89399ce90ec137fa20105ec8a8688cfc4a6ce13a4a: Status 404 returned error can't find the container with id e20c85a5e2e0affc6160bd89399ce90ec137fa20105ec8a8688cfc4a6ce13a4a
Sep 17 17:56:56.838900 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.838883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k6fsk"
Sep 17 17:56:56.845659 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.845631    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c047389_08f6_46f9_860e_47ca8dade973.slice/crio-53baf77e31f7709a18b6091dc1567c2fe2bda279f9de8d8432ad87cf02e831f5 WatchSource:0}: Error finding container 53baf77e31f7709a18b6091dc1567c2fe2bda279f9de8d8432ad87cf02e831f5: Status 404 returned error can't find the container with id 53baf77e31f7709a18b6091dc1567c2fe2bda279f9de8d8432ad87cf02e831f5
Sep 17 17:56:56.855632 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.855615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wt26z"
Sep 17 17:56:56.862120 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.862099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hd5cc"
Sep 17 17:56:56.866660 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:56.866486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-29btz"
Sep 17 17:56:56.871667 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:56:56.871617    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8b260d_874d_4242_88a5_7f3aa34a4126.slice/crio-4a7263a5fc1540d7066a9908da822411fbd49c3091ff30eef574ca10084234a1 WatchSource:0}: Error finding container 4a7263a5fc1540d7066a9908da822411fbd49c3091ff30eef574ca10084234a1: Status 404 returned error can't find the container with id 4a7263a5fc1540d7066a9908da822411fbd49c3091ff30eef574ca10084234a1
Sep 17 17:56:57.096015 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.095927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:57.096015 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.095996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:57.096238 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.096117    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:57.096238 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.096136    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:57.096238 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.096201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:58.096179716 +0000 UTC m=+4.939342856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:57.096238 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.096223    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:58.09621189 +0000 UTC m=+4.939375030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:57.166699 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.166667    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:57.196464 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.196433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:56:57.196687 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.196668    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:57.196753 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.196697    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:57.196753 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.196710    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:57.196861 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.196766    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:58.19674887 +0000 UTC m=+5.039911999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:57.581502 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.580781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:57.581502 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:57.580925    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:56:57.598377 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.598171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm" event={"ID":"f3c3c5be-9d14-4960-a097-2a89ebaae22a","Type":"ContainerStarted","Data":"a575b37570b602fafe3a9b51157570a3de1d0af4070e78d18bd9cf2d69b3f56d"}
Sep 17 17:56:57.603453 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.603329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4th94" event={"ID":"4e097696-45f7-4f39-a75b-0d5b5b51eb7c","Type":"ContainerStarted","Data":"65087530700d491ece3fe47c94e9bb0c5c43506fffb3cdd2176a655c441cbe40"}
Sep 17 17:56:57.610471 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.610438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal" event={"ID":"799c76a8366626239baf308b23899d1e","Type":"ContainerStarted","Data":"fa1587b292abb1c62654eeaac8dfe17abfe80b386c97f8f51db417b434e2af98"}
Sep 17 17:56:57.618883 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.618695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wt26z" event={"ID":"f52e62dd-57e7-4a52-a6d4-bfe36454a697","Type":"ContainerStarted","Data":"5c1530baad2813126284cf47ebad61143ddb2432c5993f28192565d81438144c"}
Sep 17 17:56:57.624680 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.624601    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:56 +0000 UTC" deadline="2028-05-20 18:55:26.647667327 +0000 UTC"
Sep 17 17:56:57.624680 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.624630    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14664h58m29.0230414s"
Sep 17 17:56:57.628408 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.628385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k6fsk" event={"ID":"1c047389-08f6-46f9-860e-47ca8dade973","Type":"ContainerStarted","Data":"53baf77e31f7709a18b6091dc1567c2fe2bda279f9de8d8432ad87cf02e831f5"}
Sep 17 17:56:57.634448 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.634373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bgkwf" event={"ID":"5077d071-443f-4f95-b556-2ac43e51d310","Type":"ContainerStarted","Data":"e20c85a5e2e0affc6160bd89399ce90ec137fa20105ec8a8688cfc4a6ce13a4a"}
Sep 17 17:56:57.639426 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.639306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"48945beee7f9ff4632d829999a76f28a70eba0c87aee2d45192179961b205977"}
Sep 17 17:56:57.653677 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.653614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dcqg2" event={"ID":"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df","Type":"ContainerStarted","Data":"1e2a2df5219fa1ab613274e5829c86e815c38be66756aa7938f742d5eb68b8bf"}
Sep 17 17:56:57.679881 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.679850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal" event={"ID":"bc4ad02cbeb2d0216949bfc307678343","Type":"ContainerStarted","Data":"8855f378cb325677d83e254ed702f3fd6ddfee059cb2d0aaff21b2506e74d58f"}
Sep 17 17:56:57.698520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.698351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerStarted","Data":"e64414e915479f36c0f79cdac28921447cb731c341440b1a9bf8020a3ec8391f"}
Sep 17 17:56:57.714526 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:57.714494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hd5cc" event={"ID":"3b8b260d-874d-4242-88a5-7f3aa34a4126","Type":"ContainerStarted","Data":"4a7263a5fc1540d7066a9908da822411fbd49c3091ff30eef574ca10084234a1"}
Sep 17 17:56:58.105250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.105208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:58.105432 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.105284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:58.105496 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.105435    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:58.105551 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.105500    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:00.105480561 +0000 UTC m=+6.948643702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:58.105640 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.105576    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:58.105640 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.105613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:00.105601138 +0000 UTC m=+6.948764277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:58.205826 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.205786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:56:58.206035 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.206006    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:58.206035 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.206027    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:58.206174 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.206040    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:58.206174 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.206098    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:00.206081253 +0000 UTC m=+7.049244402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:58.579648 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.579572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:56:58.579816 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.579706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:56:58.579816 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.579805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:56:58.579938 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:58.579909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:56:58.625404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.625362    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:56 +0000 UTC" deadline="2028-06-16 08:01:26.19863961 +0000 UTC"
Sep 17 17:56:58.625404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.625402    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15302h4m27.573242477s"
Sep 17 17:56:58.626969 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.626930    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:58.689399 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:58.689367    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:59.582476 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:56:59.581929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:56:59.582476 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:56:59.582063    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:00.120422 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:00.119628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:00.120422 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:00.119696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:00.120422 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.119862    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:00.120422 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.119926    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.119908663 +0000 UTC m=+10.963071803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:00.120422 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.120303    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:00.120422 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.120355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.120339173 +0000 UTC m=+10.963502304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:00.220908 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:00.220636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:00.220908 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.220853    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:00.220908 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.220874    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:00.220908 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.220885    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:00.221304 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.220947    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.220928941 +0000 UTC m=+11.064092079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:00.578945 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:00.578864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:00.579095 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.578996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:00.579095 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:00.579008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:00.579221 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:00.579117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:01.579701 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:01.579664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:01.580246 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:01.579827    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:02.580408 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:02.579758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:02.580408 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:02.579894    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:02.580408 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:02.580266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:02.580408 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:02.580360    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:03.580933 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:03.580009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:03.580933 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:03.580336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:04.154966 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:04.154251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:04.154966 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:04.154326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:04.154966 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.154474    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:04.154966 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.154535    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:12.154516474 +0000 UTC m=+18.997679604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:04.154966 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.154620    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:04.154966 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.154653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:12.154641411 +0000 UTC m=+18.997804537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:04.254740 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:04.254702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:04.254920 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.254869    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:04.254920 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.254894    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:04.254920 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.254906    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:04.255067 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.254962    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:12.254944183 +0000 UTC m=+19.098107329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:04.579111 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:04.579020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:04.579278 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.579167    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:04.579344 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:04.579024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:04.579481 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:04.579443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:05.581460 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:05.581422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:05.581911 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:05.581610    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:06.579225 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:06.579188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:06.579396 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:06.579187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:06.579396 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:06.579324    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:06.579507 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:06.579396    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:07.579708 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:07.579667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:07.580262 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:07.579837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:08.579596 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:08.579540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:08.579796 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:08.579694    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:08.579796 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:08.579752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:08.580205 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:08.579874    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:09.579730 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:09.579691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:09.579947 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:09.579838    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:10.578887 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:10.578853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:10.579050 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:10.578853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:10.579050 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:10.578973    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:10.579152 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:10.579046    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:11.579874 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:11.579842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:11.580224 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:11.579986    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:12.208996 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:12.208960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:12.209193 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:12.209018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:12.209193 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.209131    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:12.209308 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.209197    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.20917839 +0000 UTC m=+35.052341530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:12.209308 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.209131    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:12.209308 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.209294    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.2092732 +0000 UTC m=+35.052436344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:12.309375 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:12.309341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:12.309555 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.309528    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:12.309555 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.309573    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:12.309677 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.309588    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:12.309677 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.309655    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.30963562 +0000 UTC m=+35.152798762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:12.580002 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:12.579920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:12.580002 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:12.579964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:12.580634 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.580071    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:12.580634 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:12.580152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:13.579900 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:13.579851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:13.580135 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:13.580003    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:14.579713 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:14.579673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:14.579713 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:14.579698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:14.579926 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:14.579817    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:14.579980 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:14.579933    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:15.580453 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:15.580420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:15.580829 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:15.580548    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:16.579238 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.578915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:16.579337 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.578957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:16.579337 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:16.579273    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:16.579412 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:16.579357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:16.771921 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.771884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4th94" event={"ID":"4e097696-45f7-4f39-a75b-0d5b5b51eb7c","Type":"ContainerStarted","Data":"5265d65209054b64698e17d1bf560a1d5c7ae55b4d4b5b13af75f97c9f69711d"}
Sep 17 17:57:16.773544 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.773519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal" event={"ID":"799c76a8366626239baf308b23899d1e","Type":"ContainerStarted","Data":"6c29da286da223e06de1f455dc347ecdbab99bc35a08c0c2ab88b142401b3c50"}
Sep 17 17:57:16.775250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.775223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wt26z" event={"ID":"f52e62dd-57e7-4a52-a6d4-bfe36454a697","Type":"ContainerStarted","Data":"4992c9e6045e51ec1ae5e7b39103e57c2423ae7f4c5e5f13d583f8837ec7a860"}
Sep 17 17:57:16.776895 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.776869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k6fsk" event={"ID":"1c047389-08f6-46f9-860e-47ca8dade973","Type":"ContainerStarted","Data":"7c7864874fa9c1d6fb251ff149b31fcc3ee2567ac2680e5cc1df3c1bc7632335"}
Sep 17 17:57:16.778674 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.778646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bgkwf" event={"ID":"5077d071-443f-4f95-b556-2ac43e51d310","Type":"ContainerStarted","Data":"d5c690c3c20c191f56e06ff8064c17f3be9ef03dc4f0f71299efeabca084acbd"}
Sep 17 17:57:16.782142 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.782119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"2b3e0f34a65169043eb8e8bdcd16ad7ba8a87a91a5093768e98fcba9f2209e55"}
Sep 17 17:57:16.782330 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.782305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"6ed0aa0a1353864299e6cdb91ef7e2c8feb356811e08518dc240cd0a5866371a"}
Sep 17 17:57:16.782435 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.782365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"1031dc87cc54faa4e5096af8f3cba74a32782681171c22ee9e737a4513fde290"}
Sep 17 17:57:16.782435 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.782412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"9da99a0a53c28961eac8f621fce1ec0d429695b29c4d91e79fb64a3009bb5923"}
Sep 17 17:57:16.782530 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.782459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"486cd81fa66029d5814f50f68521099a7e36d320bd923384171b0effeb749754"}
Sep 17 17:57:16.782530 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.782501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"fc682106c0f87328a83fc9da5eaf3c077b29a4a61de4e9b3902ccf33d449ed81"}
Sep 17 17:57:16.783784 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.783759    2577 generic.go:356] "Generic (PLEG): container finished" podID="bc4ad02cbeb2d0216949bfc307678343" containerID="858d08127c19c0d43042e460007951268f419a71275372c5c89f5781f8cbd03b" exitCode=0
Sep 17 17:57:16.783884 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.783844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal" event={"ID":"bc4ad02cbeb2d0216949bfc307678343","Type":"ContainerDied","Data":"858d08127c19c0d43042e460007951268f419a71275372c5c89f5781f8cbd03b"}
Sep 17 17:57:16.785820 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.785798    2577 generic.go:356] "Generic (PLEG): container finished" podID="977fa5b8-3f32-4f08-98f1-e8de6c3b4927" containerID="e780a172242b29b91e8d89b3fb9dd37a3f76dddeca6a389296885781f94bb886" exitCode=0
Sep 17 17:57:16.785889 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.785872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerDied","Data":"e780a172242b29b91e8d89b3fb9dd37a3f76dddeca6a389296885781f94bb886"}
Sep 17 17:57:16.787900 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.787804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hd5cc" event={"ID":"3b8b260d-874d-4242-88a5-7f3aa34a4126","Type":"ContainerStarted","Data":"307d69d91fb4f651aacd5d544b4a112415338401b64bfd5d8762ca5527c32a81"}
Sep 17 17:57:16.790103 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.790072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm" event={"ID":"f3c3c5be-9d14-4960-a097-2a89ebaae22a","Type":"ContainerStarted","Data":"0344c99693ac407d1e9e3a2118579a18f8befd1a231c3a683fd3a6679d4a733d"}
Sep 17 17:57:16.818363 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.818052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-209.ec2.internal" podStartSLOduration=20.818032811 podStartE2EDuration="20.818032811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.817493359 +0000 UTC m=+23.660656509" watchObservedRunningTime="2026-09-17 17:57:16.818032811 +0000 UTC m=+23.661195960"
Sep 17 17:57:16.818529 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.818491    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4th94" podStartSLOduration=2.864778899 podStartE2EDuration="21.818482171s" totalImagesPullingTime="18.953703272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.794005649 +0000 UTC m=+23.637168798" watchObservedRunningTime="2026-09-17 17:57:16.818482171 +0000 UTC m=+23.661645319"
Sep 17 17:57:16.836558 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.836425    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k6fsk" podStartSLOduration=2.969460123 podStartE2EDuration="21.836403961s" totalImagesPullingTime="18.866943838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.836196487 +0000 UTC m=+23.679359635" watchObservedRunningTime="2026-09-17 17:57:16.836403961 +0000 UTC m=+23.679567110"
Sep 17 17:57:16.864072 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.864022    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wt26z" podStartSLOduration=2.988656616 podStartE2EDuration="21.864005983s" totalImagesPullingTime="18.875349367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.863375733 +0000 UTC m=+23.706538882" watchObservedRunningTime="2026-09-17 17:57:16.864005983 +0000 UTC m=+23.707169178"
Sep 17 17:57:16.888606 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.888543    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hd5cc" podStartSLOduration=2.989670473 podStartE2EDuration="21.888525705s" totalImagesPullingTime="18.898855232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.888195463 +0000 UTC m=+23.731358623" watchObservedRunningTime="2026-09-17 17:57:16.888525705 +0000 UTC m=+23.731688855"
Sep 17 17:57:16.967408 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:16.967353    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bgkwf" podStartSLOduration=3.086552817 podStartE2EDuration="21.967337372s" totalImagesPullingTime="18.880784555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.966774984 +0000 UTC m=+23.809938132" watchObservedRunningTime="2026-09-17 17:57:16.967337372 +0000 UTC m=+23.810500520"
Sep 17 17:57:17.177240 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.177216    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:57:17.552310 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.552204    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:57:17.177236481Z","UUID":"8753d94b-c6fa-480d-8387-6d817bd62b64","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:57:17.554735 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.554476    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:57:17.554735 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.554513    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:57:17.579399 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.579376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:17.579547 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:17.579496    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:17.795469 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.795069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dcqg2" event={"ID":"3f0b7652-f879-4ac8-a2a6-a7cf4042c0df","Type":"ContainerStarted","Data":"0900d50a34cce4f70936c47733e64072ac31ab34751e4f8d520db554663c18c4"}
Sep 17 17:57:17.797489 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.797432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal" event={"ID":"bc4ad02cbeb2d0216949bfc307678343","Type":"ContainerStarted","Data":"808428b87513e284199a0fa9a83b4f685c27efa622e79b072beadca60f61386b"}
Sep 17 17:57:17.799833 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.799807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm" event={"ID":"f3c3c5be-9d14-4960-a097-2a89ebaae22a","Type":"ContainerStarted","Data":"dc56906654dc8bbbb4be37df78a526ea4056851c05445eb65356a08a1e3129cb"}
Sep 17 17:57:17.836754 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.836658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-209.ec2.internal" podStartSLOduration=21.836643083 podStartE2EDuration="21.836643083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:17.836585897 +0000 UTC m=+24.679749047" watchObservedRunningTime="2026-09-17 17:57:17.836643083 +0000 UTC m=+24.679806232"
Sep 17 17:57:17.836905 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:17.836866    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dcqg2" podStartSLOduration=3.8996881549999998 podStartE2EDuration="22.836857381s" totalImagesPullingTime="18.937169226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:17.816372778 +0000 UTC m=+24.659535928" watchObservedRunningTime="2026-09-17 17:57:17.836857381 +0000 UTC m=+24.680020529"
Sep 17 17:57:18.579597 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:18.579528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:18.579597 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:18.579528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:18.579796 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:18.579701    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:18.579839 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:18.579815    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:18.804769 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:18.804731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm" event={"ID":"f3c3c5be-9d14-4960-a097-2a89ebaae22a","Type":"ContainerStarted","Data":"18eff8c6b12a4b014bfe723f33654b56b03a539469969111e05df1491daf4061"}
Sep 17 17:57:18.808877 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:18.808840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"5dc774042eda6d4043daed01bc63c0fe66c3b1304df9f5c93cf4b16bffa02e73"}
Sep 17 17:57:18.837750 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:18.837641    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z8xm" podStartSLOduration=3.273254117 podStartE2EDuration="23.837617695s" totalImagesPullingTime="20.564363578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:18.837589527 +0000 UTC m=+25.680752677" watchObservedRunningTime="2026-09-17 17:57:18.837617695 +0000 UTC m=+25.680780834"
Sep 17 17:57:19.269897 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:19.269862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:57:19.270547 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:19.270519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:57:19.579475 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:19.579396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:19.579659 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:19.579528    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:19.811688 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:19.811655    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:57:19.812279 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:19.812141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bgkwf"
Sep 17 17:57:20.579230 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:20.579198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:20.579401 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:20.579207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:20.579401 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:20.579312    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:20.579490 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:20.579416    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:21.579443 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:21.579271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:21.580116 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:21.579522    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:21.817754 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:21.817717    2577 generic.go:356] "Generic (PLEG): container finished" podID="977fa5b8-3f32-4f08-98f1-e8de6c3b4927" containerID="76b3a8c424afcb3d654f6cc419d8b3bbb87963f462511260a507625d5c32ed30" exitCode=0
Sep 17 17:57:21.817913 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:21.817782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerDied","Data":"76b3a8c424afcb3d654f6cc419d8b3bbb87963f462511260a507625d5c32ed30"}
Sep 17 17:57:21.824516 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:21.824491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" event={"ID":"1e73f216-d440-41b0-a209-65eefbe31903","Type":"ContainerStarted","Data":"72ba015bde6be78fb791eaa2f80caad8bdfc0bf77c65b3082d7315cfcfb29cc9"}
Sep 17 17:57:21.873316 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:21.873200    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv" podStartSLOduration=7.634423981 podStartE2EDuration="26.873185229s" totalImagesPullingTime="19.238761248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:21.872146755 +0000 UTC m=+28.715309903" watchObservedRunningTime="2026-09-17 17:57:21.873185229 +0000 UTC m=+28.716348377"
Sep 17 17:57:22.578952 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.578925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:22.579064 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.578933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:22.579117 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:22.579057    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:22.579166 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:22.579144    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:22.806954 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.806709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4qs8"]
Sep 17 17:57:22.809282 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.809254    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9jfv"]
Sep 17 17:57:22.809938 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.809907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9flf9"]
Sep 17 17:57:22.810048 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.810016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:22.810139 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:22.810116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:22.829203 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.829171    2577 generic.go:356] "Generic (PLEG): container finished" podID="977fa5b8-3f32-4f08-98f1-e8de6c3b4927" containerID="88a6ad36ade2991465502047dd3f8ddc96583607151f3801f6e29615c0626dc0" exitCode=0
Sep 17 17:57:22.829353 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.829262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerDied","Data":"88a6ad36ade2991465502047dd3f8ddc96583607151f3801f6e29615c0626dc0"}
Sep 17 17:57:22.829353 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.829310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:22.829456 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.829310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:22.829456 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:22.829422    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:22.830008 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:22.829978    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:22.830008 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.830003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:57:22.830210 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.830085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:57:22.830210 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.830136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:57:22.849544 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.849512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:57:22.849718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:22.849681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:57:23.834068 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:23.833978    2577 generic.go:356] "Generic (PLEG): container finished" podID="977fa5b8-3f32-4f08-98f1-e8de6c3b4927" containerID="30e4df3ee5357fda126e5e867f6364aca2bfc2e1200a7d98ff728fe7bc192a12" exitCode=0
Sep 17 17:57:23.834509 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:23.834057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerDied","Data":"30e4df3ee5357fda126e5e867f6364aca2bfc2e1200a7d98ff728fe7bc192a12"}
Sep 17 17:57:24.579844 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:24.579812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:24.580023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:24.579846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:24.580023 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:24.579846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:24.580023 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:24.579930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:24.580157 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:24.580029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:24.580157 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:24.580100    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:26.579235 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:26.579199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:26.579662 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:26.579239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:26.579662 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:26.579199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:26.579662 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:26.579326    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:26.579662 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:26.579371    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:26.579662 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:26.579438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:28.225052 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.225014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:28.225518 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.225081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:28.225518 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.225162    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:28.225518 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.225181    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:28.225518 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.225225    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret podName:4acb9960-a709-42b3-9f79-ef953d15e558 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.225210603 +0000 UTC m=+67.068373734 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret") pod "global-pull-secret-syncer-j4qs8" (UID: "4acb9960-a709-42b3-9f79-ef953d15e558") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:28.225518 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.225240    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.22523316 +0000 UTC m=+67.068396285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:28.325836 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.325811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:28.325991 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.325969    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:28.326035 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.325999    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:28.326035 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.326010    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8bn4p for pod openshift-network-diagnostics/network-check-target-b9jfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:28.326095 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.326060    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p podName:de04cd13-f4a7-4f10-a014-7b98b6736e11 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.326046762 +0000 UTC m=+67.169209888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bn4p" (UniqueName: "kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p") pod "network-check-target-b9jfv" (UID: "de04cd13-f4a7-4f10-a014-7b98b6736e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:28.579844 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.579811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:28.580055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.579870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:28.580055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.579970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:28.580055 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.579968    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9jfv" podUID="de04cd13-f4a7-4f10-a014-7b98b6736e11"
Sep 17 17:57:28.580055 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.580035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9flf9" podUID="761dc4fd-5005-48a1-9874-bfa11455f308"
Sep 17 17:57:28.580209 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:28.580098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4qs8" podUID="4acb9960-a709-42b3-9f79-ef953d15e558"
Sep 17 17:57:28.847380 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.847349    2577 generic.go:356] "Generic (PLEG): container finished" podID="977fa5b8-3f32-4f08-98f1-e8de6c3b4927" containerID="5c17674cfda3b45baef493da7539d877af9442222b8e93fce082f5336b44a1e2" exitCode=0
Sep 17 17:57:28.847528 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:28.847389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerDied","Data":"5c17674cfda3b45baef493da7539d877af9442222b8e93fce082f5336b44a1e2"}
Sep 17 17:57:29.855978 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.855749    2577 generic.go:356] "Generic (PLEG): container finished" podID="977fa5b8-3f32-4f08-98f1-e8de6c3b4927" containerID="9585b4e82ecca09b20a1bce11965bc172077fac4b474d373a22cc63c0dcbf1b4" exitCode=0
Sep 17 17:57:29.855978 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.855825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerDied","Data":"9585b4e82ecca09b20a1bce11965bc172077fac4b474d373a22cc63c0dcbf1b4"}
Sep 17 17:57:29.868255 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.868233    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-209.ec2.internal" event="NodeReady"
Sep 17 17:57:29.868381 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.868349    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:57:29.938714 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.938691    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ssns9"]
Sep 17 17:57:29.965185 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.965162    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s5qlj"]
Sep 17 17:57:29.965291 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.965274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:29.968881 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.968852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5ghb8\""
Sep 17 17:57:29.969573 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.969543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:57:29.969739 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.969718    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:57:29.970668 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.970653    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:57:29.981407 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.981387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:29.981407 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.981400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ssns9"]
Sep 17 17:57:29.981546 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.981431    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s5qlj"]
Sep 17 17:57:29.986328 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.986299    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zr8ck\""
Sep 17 17:57:29.986539 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.986522    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:57:29.986941 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:29.986922    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:57:30.041038 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.041005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33d54a6-6442-4f25-84eb-aad9b3c40c0d-cert\") pod \"ingress-canary-ssns9\" (UID: \"f33d54a6-6442-4f25-84eb-aad9b3c40c0d\") " pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.041182 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.041044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmzr\" (UniqueName: \"kubernetes.io/projected/f33d54a6-6442-4f25-84eb-aad9b3c40c0d-kube-api-access-pbmzr\") pod \"ingress-canary-ssns9\" (UID: \"f33d54a6-6442-4f25-84eb-aad9b3c40c0d\") " pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.142354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.142281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmzr\" (UniqueName: \"kubernetes.io/projected/f33d54a6-6442-4f25-84eb-aad9b3c40c0d-kube-api-access-pbmzr\") pod \"ingress-canary-ssns9\" (UID: \"f33d54a6-6442-4f25-84eb-aad9b3c40c0d\") " pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.142354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.142330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-config-volume\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.142354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.142351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-tmp-dir\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.142547 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.142410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33d54a6-6442-4f25-84eb-aad9b3c40c0d-cert\") pod \"ingress-canary-ssns9\" (UID: \"f33d54a6-6442-4f25-84eb-aad9b3c40c0d\") " pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.142547 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.142430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhl7\" (UniqueName: \"kubernetes.io/projected/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-kube-api-access-9bhl7\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.142547 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.142450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.146725 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.146695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33d54a6-6442-4f25-84eb-aad9b3c40c0d-cert\") pod \"ingress-canary-ssns9\" (UID: \"f33d54a6-6442-4f25-84eb-aad9b3c40c0d\") " pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.151968 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.151944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmzr\" (UniqueName: \"kubernetes.io/projected/f33d54a6-6442-4f25-84eb-aad9b3c40c0d-kube-api-access-pbmzr\") pod \"ingress-canary-ssns9\" (UID: \"f33d54a6-6442-4f25-84eb-aad9b3c40c0d\") " pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.243077 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.243039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhl7\" (UniqueName: \"kubernetes.io/projected/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-kube-api-access-9bhl7\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.243258 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.243083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.243258 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.243125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-config-volume\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.243258 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.243152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-tmp-dir\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.243415 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:30.243263    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:30.243415 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:30.243327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:30.743310434 +0000 UTC m=+37.586473565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:30.243534 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.243517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-tmp-dir\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.243744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.243725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-config-volume\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.253867 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.253847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhl7\" (UniqueName: \"kubernetes.io/projected/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-kube-api-access-9bhl7\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.274933 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.274901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ssns9"
Sep 17 17:57:30.579725 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.579693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:57:30.579725 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.579717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:57:30.579974 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.579693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:57:30.584516 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.584496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:30.584650 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.584643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:30.584819 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.584806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7697\""
Sep 17 17:57:30.584977 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.584963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:30.585037 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.584999    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6psft\""
Sep 17 17:57:30.585356 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.585341    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:30.613890 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.613870    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ssns9"]
Sep 17 17:57:30.617752 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:57:30.617732    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33d54a6_6442_4f25_84eb_aad9b3c40c0d.slice/crio-7d9d3a09bb183c8215c18d2ab023c3015359dd1160f21527d12b2cb7a849620f WatchSource:0}: Error finding container 7d9d3a09bb183c8215c18d2ab023c3015359dd1160f21527d12b2cb7a849620f: Status 404 returned error can't find the container with id 7d9d3a09bb183c8215c18d2ab023c3015359dd1160f21527d12b2cb7a849620f
Sep 17 17:57:30.747686 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.747652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:30.747881 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:30.747778    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:30.747881 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:30.747836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:31.747819724 +0000 UTC m=+38.590982855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:30.858946 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.858870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ssns9" event={"ID":"f33d54a6-6442-4f25-84eb-aad9b3c40c0d","Type":"ContainerStarted","Data":"7d9d3a09bb183c8215c18d2ab023c3015359dd1160f21527d12b2cb7a849620f"}
Sep 17 17:57:30.861974 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.861953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29btz" event={"ID":"977fa5b8-3f32-4f08-98f1-e8de6c3b4927","Type":"ContainerStarted","Data":"c41c03f5bf80a6182ec392a06563d770a3e545136896d9092a608ce3c7a9fdab"}
Sep 17 17:57:30.886530 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:30.886488    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-29btz" podStartSLOduration=7.302025343 podStartE2EDuration="35.8864743s" totalImagesPullingTime="28.584448957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:30.885416814 +0000 UTC m=+37.728579962" watchObservedRunningTime="2026-09-17 17:57:30.8864743 +0000 UTC m=+37.729637448"
Sep 17 17:57:31.756436 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:31.756398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:31.756629 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:31.756603    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:31.756701 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:31.756691    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:33.756668764 +0000 UTC m=+40.599831892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:33.774029 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:33.773838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:33.774410 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:33.773989    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:33.774410 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:33.774132    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:37.774118512 +0000 UTC m=+44.617281652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:33.870858 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:33.870823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ssns9" event={"ID":"f33d54a6-6442-4f25-84eb-aad9b3c40c0d","Type":"ContainerStarted","Data":"e8e8229801902b0ca2f22e7d57da04a2b19c39e415b700d7d465ffdd83fe2fa6"}
Sep 17 17:57:33.892294 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:33.892248    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ssns9" podStartSLOduration=2.564503213 podStartE2EDuration="4.892233322s" totalImagesPullingTime="2.327730109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:33.891835653 +0000 UTC m=+40.734998800" watchObservedRunningTime="2026-09-17 17:57:33.892233322 +0000 UTC m=+40.735396470"
Sep 17 17:57:37.802062 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:37.802024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:37.802509 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:37.802144    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:37.802509 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:37.802196    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:45.802183357 +0000 UTC m=+52.645346482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:45.851535 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:45.851494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:57:45.851996 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:45.851638    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:45.851996 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:45.851689    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:01.851676215 +0000 UTC m=+68.694839341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:54.846316 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:54.846289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlxrv"
Sep 17 17:57:56.921172 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.921137    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"]
Sep 17 17:57:56.923916 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.923888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:56.930325 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.929951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:57:56.930325 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.929962    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:57:56.930325 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.930257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t2qvf\""
Sep 17 17:57:56.930490 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.930441    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:57:56.930718 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.930592    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:57:56.930934 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.930917    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cc8c845db-rbdqs"]
Sep 17 17:57:56.932672 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.932658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:56.937525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.937509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:57:56.937688 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.937546    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:57:56.937688 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.937622    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:57:56.937688 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.937635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:57:56.937957 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.937858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:57:56.937957 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.937905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:57:56.940519 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.940501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"]
Sep 17 17:57:56.943038 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.943019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c47l4\""
Sep 17 17:57:56.952271 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:56.952250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cc8c845db-rbdqs"]
Sep 17 17:57:57.027306 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.027270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.027488 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.027347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5e8a92fe-3574-4410-9565-d7926dc55b83-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.027488 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.027376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfpw\" (UniqueName: \"kubernetes.io/projected/5e8a92fe-3574-4410-9565-d7926dc55b83-kube-api-access-6lfpw\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.029443 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.029420    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cjccs"]
Sep 17 17:57:57.031354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.031340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.036169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:57:57.036319 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.036298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hbtkh\""
Sep 17 17:57:57.036319 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.036303    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:57.036475 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.036454    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:57.036952 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.036933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:57:57.043320 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.043302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:57:57.050735 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.050711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cjccs"]
Sep 17 17:57:57.128681 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vbl\" (UniqueName: \"kubernetes.io/projected/dcd70886-5382-4843-807a-b4c6b7cffa45-kube-api-access-j9vbl\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.128681 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-config\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.128914 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-stats-auth\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.128914 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.128914 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.128914 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.128821    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:57:57.128914 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.128877    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls podName:5e8a92fe-3574-4410-9565-d7926dc55b83 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:57.628861693 +0000 UTC m=+64.472024819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-w49ts" (UID: "5e8a92fe-3574-4410-9565-d7926dc55b83") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-serving-cert\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.128984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.129020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv879\" (UniqueName: \"kubernetes.io/projected/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-kube-api-access-rv879\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.129078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5e8a92fe-3574-4410-9565-d7926dc55b83-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.129106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfpw\" (UniqueName: \"kubernetes.io/projected/5e8a92fe-3574-4410-9565-d7926dc55b83-kube-api-access-6lfpw\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.129135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-default-certificate\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.129171 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.129161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-trusted-ca\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.129716 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.129699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5e8a92fe-3574-4410-9565-d7926dc55b83-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.145093 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.145070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfpw\" (UniqueName: \"kubernetes.io/projected/5e8a92fe-3574-4410-9565-d7926dc55b83-kube-api-access-6lfpw\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.230201 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv879\" (UniqueName: \"kubernetes.io/projected/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-kube-api-access-rv879\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.230356 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-default-certificate\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.230356 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-trusted-ca\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.230356 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vbl\" (UniqueName: \"kubernetes.io/projected/dcd70886-5382-4843-807a-b4c6b7cffa45-kube-api-access-j9vbl\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.230516 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-config\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.230516 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-stats-auth\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.230665 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.230665 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-serving-cert\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.230665 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.230612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.230810 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.230665    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:57:57.230810 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.230758    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:57.730728104 +0000 UTC m=+64.573891233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : secret "router-metrics-certs-default" not found
Sep 17 17:57:57.230810 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.230779    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:57.730770161 +0000 UTC m=+64.573933320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : configmap references non-existent config key: service-ca.crt
Sep 17 17:57:57.231182 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.231156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-config\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.231477 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.231448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-trusted-ca\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.232783 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.232763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-default-certificate\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.232879 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.232827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-serving-cert\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.232926 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.232907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-stats-auth\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.242146 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.242125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vbl\" (UniqueName: \"kubernetes.io/projected/dcd70886-5382-4843-807a-b4c6b7cffa45-kube-api-access-j9vbl\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.243135 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.243116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv879\" (UniqueName: \"kubernetes.io/projected/a8ef87c2-1676-4f2d-ae74-a952d8f18b14-kube-api-access-rv879\") pod \"console-operator-fd4bb46c-cjccs\" (UID: \"a8ef87c2-1676-4f2d-ae74-a952d8f18b14\") " pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.343073 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.343032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:57:57.620372 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.620293    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cjccs"]
Sep 17 17:57:57.623897 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:57:57.623869    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ef87c2_1676_4f2d_ae74_a952d8f18b14.slice/crio-2d46e650f502a1fe9e948e44f8b0138ceb824b3fc8b2a8eb637daa1d4ec22d04 WatchSource:0}: Error finding container 2d46e650f502a1fe9e948e44f8b0138ceb824b3fc8b2a8eb637daa1d4ec22d04: Status 404 returned error can't find the container with id 2d46e650f502a1fe9e948e44f8b0138ceb824b3fc8b2a8eb637daa1d4ec22d04
Sep 17 17:57:57.634189 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.634167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:57.634306 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.634291    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:57:57.634348 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.634343    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls podName:5e8a92fe-3574-4410-9565-d7926dc55b83 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.63432835 +0000 UTC m=+65.477491476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-w49ts" (UID: "5e8a92fe-3574-4410-9565-d7926dc55b83") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:57:57.734746 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.734718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.734896 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.734778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:57.734896 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.734822    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:57:57.734971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.734903    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.734887394 +0000 UTC m=+65.578050524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : secret "router-metrics-certs-default" not found
Sep 17 17:57:57.734971 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:57.734931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.734917909 +0000 UTC m=+65.578081035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : configmap references non-existent config key: service-ca.crt
Sep 17 17:57:57.831363 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.831333    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"]
Sep 17 17:57:57.834180 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.834164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:57.837250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.837223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2wbbg\""
Sep 17 17:57:57.837372 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.837223    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:57.837372 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.837339    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:57:57.837372 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.837346    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:57.837510 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.837480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:57:57.846093 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.846075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"]
Sep 17 17:57:57.928960 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.928919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" event={"ID":"a8ef87c2-1676-4f2d-ae74-a952d8f18b14","Type":"ContainerStarted","Data":"2d46e650f502a1fe9e948e44f8b0138ceb824b3fc8b2a8eb637daa1d4ec22d04"}
Sep 17 17:57:57.936242 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.936221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cbb56b5-5062-4840-8036-e7a2ca11f98a-config\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:57.936300 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.936271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cbb56b5-5062-4840-8036-e7a2ca11f98a-serving-cert\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:57.936300 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:57.936287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkmj\" (UniqueName: \"kubernetes.io/projected/8cbb56b5-5062-4840-8036-e7a2ca11f98a-kube-api-access-6gkmj\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.037215 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.037183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cbb56b5-5062-4840-8036-e7a2ca11f98a-config\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.037363 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.037240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cbb56b5-5062-4840-8036-e7a2ca11f98a-serving-cert\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.037363 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.037265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkmj\" (UniqueName: \"kubernetes.io/projected/8cbb56b5-5062-4840-8036-e7a2ca11f98a-kube-api-access-6gkmj\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.037766 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.037749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cbb56b5-5062-4840-8036-e7a2ca11f98a-config\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.039486 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.039469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cbb56b5-5062-4840-8036-e7a2ca11f98a-serving-cert\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.047145 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.047123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkmj\" (UniqueName: \"kubernetes.io/projected/8cbb56b5-5062-4840-8036-e7a2ca11f98a-kube-api-access-6gkmj\") pod \"service-ca-operator-57cdc4b768-rnd92\" (UID: \"8cbb56b5-5062-4840-8036-e7a2ca11f98a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.142503 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.142468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"
Sep 17 17:57:58.420233 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.420202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92"]
Sep 17 17:57:58.423952 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:57:58.423915    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbb56b5_5062_4840_8036_e7a2ca11f98a.slice/crio-4c2320bceacde8c28fea1cc1421bc6f710e5744ff90ec837721784825b753e49 WatchSource:0}: Error finding container 4c2320bceacde8c28fea1cc1421bc6f710e5744ff90ec837721784825b753e49: Status 404 returned error can't find the container with id 4c2320bceacde8c28fea1cc1421bc6f710e5744ff90ec837721784825b753e49
Sep 17 17:57:58.644024 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.643942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:57:58.644184 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:58.644112    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:57:58.644232 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:58.644201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls podName:5e8a92fe-3574-4410-9565-d7926dc55b83 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.644179339 +0000 UTC m=+67.487342469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-w49ts" (UID: "5e8a92fe-3574-4410-9565-d7926dc55b83") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:57:58.744861 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.744821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:58.745046 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.744890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:57:58.745046 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:58.744935    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:57:58.745046 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:58.745000    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.744981189 +0000 UTC m=+67.588144331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : secret "router-metrics-certs-default" not found
Sep 17 17:57:58.745046 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:57:58.745016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:00.745008897 +0000 UTC m=+67.588172023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : configmap references non-existent config key: service-ca.crt
Sep 17 17:57:58.933514 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:57:58.933461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92" event={"ID":"8cbb56b5-5062-4840-8036-e7a2ca11f98a","Type":"ContainerStarted","Data":"4c2320bceacde8c28fea1cc1421bc6f710e5744ff90ec837721784825b753e49"}
Sep 17 17:58:00.257063 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.257029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:58:00.257382 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.257125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:58:00.262379 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.262362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:58:00.262444 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.262392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:58:00.267585 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.267539    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:58:00.267705 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.267635    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs podName:761dc4fd-5005-48a1-9874-bfa11455f308 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:04.267615462 +0000 UTC m=+131.110778609 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs") pod "network-metrics-daemon-9flf9" (UID: "761dc4fd-5005-48a1-9874-bfa11455f308") : secret "metrics-daemon-secret" not found
Sep 17 17:58:00.269752 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.269723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4acb9960-a709-42b3-9f79-ef953d15e558-original-pull-secret\") pod \"global-pull-secret-syncer-j4qs8\" (UID: \"4acb9960-a709-42b3-9f79-ef953d15e558\") " pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:58:00.289742 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.289679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4qs8"
Sep 17 17:58:00.357581 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.357533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:58:00.360781 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.360732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.371992 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.371969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.381228 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.381203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bn4p\" (UniqueName: \"kubernetes.io/projected/de04cd13-f4a7-4f10-a014-7b98b6736e11-kube-api-access-8bn4p\") pod \"network-check-target-b9jfv\" (UID: \"de04cd13-f4a7-4f10-a014-7b98b6736e11\") " pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:58:00.604782 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.604700    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6psft\""
Sep 17 17:58:00.610933 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.610898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:58:00.659396 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.659328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:58:00.659554 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.659506    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:00.659632 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.659597    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls podName:5e8a92fe-3574-4410-9565-d7926dc55b83 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.659578895 +0000 UTC m=+71.502742041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-w49ts" (UID: "5e8a92fe-3574-4410-9565-d7926dc55b83") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:00.760746 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.760554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:00.760746 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.760644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:00.760746 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.760714    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:58:00.761007 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.760784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.760763629 +0000 UTC m=+71.603926772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : secret "router-metrics-certs-default" not found
Sep 17 17:58:00.761007 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:00.760801    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.760793638 +0000 UTC m=+71.603956764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : configmap references non-existent config key: service-ca.crt
Sep 17 17:58:00.940000 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.939960    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/0.log"
Sep 17 17:58:00.940187 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.940004    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8ef87c2-1676-4f2d-ae74-a952d8f18b14" containerID="0138144dd9e87bb43c28f69f31aea359591cc509fc844c7c28ccef159acf27ed" exitCode=255
Sep 17 17:58:00.940187 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.940085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" event={"ID":"a8ef87c2-1676-4f2d-ae74-a952d8f18b14","Type":"ContainerDied","Data":"0138144dd9e87bb43c28f69f31aea359591cc509fc844c7c28ccef159acf27ed"}
Sep 17 17:58:00.940382 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.940366    2577 scope.go:117] "RemoveContainer" containerID="0138144dd9e87bb43c28f69f31aea359591cc509fc844c7c28ccef159acf27ed"
Sep 17 17:58:00.941648 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.941627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92" event={"ID":"8cbb56b5-5062-4840-8036-e7a2ca11f98a","Type":"ContainerStarted","Data":"e79430ff2c2cb295ee54be7d333d65ccc79e0eb59a8fd4b58d8e3a2a07117130"}
Sep 17 17:58:00.987115 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:00.987063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92" podStartSLOduration=1.639976103 podStartE2EDuration="3.987042796s" totalImagesPullingTime="2.347066693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:00.986461772 +0000 UTC m=+67.829624922" watchObservedRunningTime="2026-09-17 17:58:00.987042796 +0000 UTC m=+67.830205942"
Sep 17 17:58:01.028327 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.028276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9jfv"]
Sep 17 17:58:01.029678 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.029616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4qs8"]
Sep 17 17:58:01.033416 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:01.033388    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acb9960_a709_42b3_9f79_ef953d15e558.slice/crio-90c504714c0d56c18c477e5a35e9710b14ed46c9550dc38c9551687c5ddc3837 WatchSource:0}: Error finding container 90c504714c0d56c18c477e5a35e9710b14ed46c9550dc38c9551687c5ddc3837: Status 404 returned error can't find the container with id 90c504714c0d56c18c477e5a35e9710b14ed46c9550dc38c9551687c5ddc3837
Sep 17 17:58:01.034106 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:01.034069    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde04cd13_f4a7_4f10_a014_7b98b6736e11.slice/crio-7da2809b2bf84bd0f485aec09e2baaa6d77eed701e3bd3abaa4b374453f02c7d WatchSource:0}: Error finding container 7da2809b2bf84bd0f485aec09e2baaa6d77eed701e3bd3abaa4b374453f02c7d: Status 404 returned error can't find the container with id 7da2809b2bf84bd0f485aec09e2baaa6d77eed701e3bd3abaa4b374453f02c7d
Sep 17 17:58:01.641916 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.641867    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"]
Sep 17 17:58:01.644466 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.644433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"
Sep 17 17:58:01.648285 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.648262    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:01.649032 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.648941    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:58:01.650015 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.649991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2f8lx\""
Sep 17 17:58:01.668715 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.668688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdtz\" (UniqueName: \"kubernetes.io/projected/b6e91b81-c4f3-4032-8fc9-4cdf595b52f3-kube-api-access-qvdtz\") pod \"migrator-76794cbd6c-lwv2h\" (UID: \"b6e91b81-c4f3-4032-8fc9-4cdf595b52f3\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"
Sep 17 17:58:01.685425 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.685349    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"]
Sep 17 17:58:01.770137 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.770095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdtz\" (UniqueName: \"kubernetes.io/projected/b6e91b81-c4f3-4032-8fc9-4cdf595b52f3-kube-api-access-qvdtz\") pod \"migrator-76794cbd6c-lwv2h\" (UID: \"b6e91b81-c4f3-4032-8fc9-4cdf595b52f3\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"
Sep 17 17:58:01.787367 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.787303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdtz\" (UniqueName: \"kubernetes.io/projected/b6e91b81-c4f3-4032-8fc9-4cdf595b52f3-kube-api-access-qvdtz\") pod \"migrator-76794cbd6c-lwv2h\" (UID: \"b6e91b81-c4f3-4032-8fc9-4cdf595b52f3\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"
Sep 17 17:58:01.859754 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.859714    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hf8l4"]
Sep 17 17:58:01.862280 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.862255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.865764 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.865739    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:58:01.866077 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.866020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:58:01.866188 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.866153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2k5lz\""
Sep 17 17:58:01.867105 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.867082    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.870521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-crio-socket\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.870598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.870679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.870718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gs6\" (UniqueName: \"kubernetes.io/projected/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-api-access-25gs6\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.870745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-data-volume\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.870849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.870951    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:58:01.871121 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.871011    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls podName:ceb131a8-2b87-49d7-855e-2b2b89c9fa99 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:33.870992109 +0000 UTC m=+100.714155253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls") pod "dns-default-s5qlj" (UID: "ceb131a8-2b87-49d7-855e-2b2b89c9fa99") : secret "dns-default-metrics-tls" not found
Sep 17 17:58:01.875154 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.875136    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:58:01.885225 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.885183    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hf8l4"]
Sep 17 17:58:01.947641 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.947615    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 17:58:01.948128 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.948106    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/0.log"
Sep 17 17:58:01.948277 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.948152    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8ef87c2-1676-4f2d-ae74-a952d8f18b14" containerID="1a6cef62dd9a5acc9bf95861e904f0132ec155cebf672af64cdfe612ab162df1" exitCode=255
Sep 17 17:58:01.948484 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.948447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" event={"ID":"a8ef87c2-1676-4f2d-ae74-a952d8f18b14","Type":"ContainerDied","Data":"1a6cef62dd9a5acc9bf95861e904f0132ec155cebf672af64cdfe612ab162df1"}
Sep 17 17:58:01.948592 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.948507    2577 scope.go:117] "RemoveContainer" containerID="0138144dd9e87bb43c28f69f31aea359591cc509fc844c7c28ccef159acf27ed"
Sep 17 17:58:01.948592 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.948551    2577 scope.go:117] "RemoveContainer" containerID="1a6cef62dd9a5acc9bf95861e904f0132ec155cebf672af64cdfe612ab162df1"
Sep 17 17:58:01.948835 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.948790    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cjccs_openshift-console-operator(a8ef87c2-1676-4f2d-ae74-a952d8f18b14)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" podUID="a8ef87c2-1676-4f2d-ae74-a952d8f18b14"
Sep 17 17:58:01.949866 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.949839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j4qs8" event={"ID":"4acb9960-a709-42b3-9f79-ef953d15e558","Type":"ContainerStarted","Data":"90c504714c0d56c18c477e5a35e9710b14ed46c9550dc38c9551687c5ddc3837"}
Sep 17 17:58:01.951512 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.951458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9jfv" event={"ID":"de04cd13-f4a7-4f10-a014-7b98b6736e11","Type":"ContainerStarted","Data":"7da2809b2bf84bd0f485aec09e2baaa6d77eed701e3bd3abaa4b374453f02c7d"}
Sep 17 17:58:01.957035 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.956974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.971583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.971670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gs6\" (UniqueName: \"kubernetes.io/projected/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-api-access-25gs6\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.971705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-data-volume\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.971853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-crio-socket\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.971921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-crio-socket\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.971673    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.971949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.971996    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.471977021 +0000 UTC m=+69.315140162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.972020    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:01.972404 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:01.972089    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.472052668 +0000 UTC m=+69.315215794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:01.973690 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.973180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-data-volume\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:01.983199 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:01.983156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gs6\" (UniqueName: \"kubernetes.io/projected/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-api-access-25gs6\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:02.264752 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:02.264589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h"]
Sep 17 17:58:02.269373 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:02.269327    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e91b81_c4f3_4032_8fc9_4cdf595b52f3.slice/crio-1d8f43b00195c09cd47176f2e5937a8550475be17031fadab5fbb0d5bff2e635 WatchSource:0}: Error finding container 1d8f43b00195c09cd47176f2e5937a8550475be17031fadab5fbb0d5bff2e635: Status 404 returned error can't find the container with id 1d8f43b00195c09cd47176f2e5937a8550475be17031fadab5fbb0d5bff2e635
Sep 17 17:58:02.477149 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:02.477106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:02.477326 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:02.477178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:02.477326 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:02.477239    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:02.477326 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:02.477302    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:02.477326 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:02.477311    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:03.477287326 +0000 UTC m=+70.320450476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:02.477497 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:02.477357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:03.477340576 +0000 UTC m=+70.320503704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:02.956744 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:02.956707    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 17:58:02.957194 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:02.957138    2577 scope.go:117] "RemoveContainer" containerID="1a6cef62dd9a5acc9bf95861e904f0132ec155cebf672af64cdfe612ab162df1"
Sep 17 17:58:02.957443 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:02.957373    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cjccs_openshift-console-operator(a8ef87c2-1676-4f2d-ae74-a952d8f18b14)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" podUID="a8ef87c2-1676-4f2d-ae74-a952d8f18b14"
Sep 17 17:58:02.958500 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:02.958452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h" event={"ID":"b6e91b81-c4f3-4032-8fc9-4cdf595b52f3","Type":"ContainerStarted","Data":"1d8f43b00195c09cd47176f2e5937a8550475be17031fadab5fbb0d5bff2e635"}
Sep 17 17:58:03.486371 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:03.486331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:03.486541 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:03.486403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:03.486541 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:03.486473    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:03.486541 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:03.486516    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:03.486749 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:03.486555    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.486532934 +0000 UTC m=+72.329696084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:03.486749 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:03.486634    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.486617938 +0000 UTC m=+72.329781078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:04.550270 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:04.550243    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k6fsk_1c047389-08f6-46f9-860e-47ca8dade973/dns-node-resolver/0.log"
Sep 17 17:58:04.695824 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:04.695757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:58:04.696012 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:04.695923    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:04.696012 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:04.696001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls podName:5e8a92fe-3574-4410-9565-d7926dc55b83 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.695980913 +0000 UTC m=+79.539144039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-w49ts" (UID: "5e8a92fe-3574-4410-9565-d7926dc55b83") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:04.796527 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:04.796493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:04.796527 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:04.796539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:04.796793 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:04.796688    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:58:04.796793 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:04.796704    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.796690591 +0000 UTC m=+79.639853730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : configmap references non-existent config key: service-ca.crt
Sep 17 17:58:04.796793 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:04.796751    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.796734642 +0000 UTC m=+79.639897788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : secret "router-metrics-certs-default" not found
Sep 17 17:58:05.348249 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.348222    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4th94_4e097696-45f7-4f39-a75b-0d5b5b51eb7c/node-ca/0.log"
Sep 17 17:58:05.502859 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.502813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:05.503056 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:05.502937    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.503056 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:05.503011    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:09.502988557 +0000 UTC m=+76.346151695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.503056 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:05.503029    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.503223 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.502969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:05.503223 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:05.503082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:09.503065425 +0000 UTC m=+76.346228572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.968092 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.968051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j4qs8" event={"ID":"4acb9960-a709-42b3-9f79-ef953d15e558","Type":"ContainerStarted","Data":"2ab31d47d2808d7a2bdbdf8c8deab381007c56e294e6664ab2564e0cbb01dc41"}
Sep 17 17:58:05.969839 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.969809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h" event={"ID":"b6e91b81-c4f3-4032-8fc9-4cdf595b52f3","Type":"ContainerStarted","Data":"3cb2013b7498b1c6e30dfef51e69c1e20ceba205405594f3f684bc7649fac103"}
Sep 17 17:58:05.969974 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.969870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h" event={"ID":"b6e91b81-c4f3-4032-8fc9-4cdf595b52f3","Type":"ContainerStarted","Data":"8228957cff2d43c37b2f8fc2325863df2ad0c91cc12bb42738d61bed411778c1"}
Sep 17 17:58:05.971161 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.971138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9jfv" event={"ID":"de04cd13-f4a7-4f10-a014-7b98b6736e11","Type":"ContainerStarted","Data":"e81a573bc25d5859502a6cb8e8843acbac5dd06a518caef47304be666dabad57"}
Sep 17 17:58:05.971276 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.971260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:58:05.987483 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:05.987439    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j4qs8" podStartSLOduration=66.38519504 podStartE2EDuration="1m10.987424928s" totalImagesPullingTime="4.602229888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:05.986040138 +0000 UTC m=+72.829203286" watchObservedRunningTime="2026-09-17 17:58:05.987424928 +0000 UTC m=+72.830588076"
Sep 17 17:58:06.004532 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:06.004496    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b9jfv" podStartSLOduration=66.418291074 podStartE2EDuration="1m11.004484108s" totalImagesPullingTime="4.586193034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:06.003814401 +0000 UTC m=+72.846977550" watchObservedRunningTime="2026-09-17 17:58:06.004484108 +0000 UTC m=+72.847647255"
Sep 17 17:58:06.022673 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:06.022632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-lwv2h" podStartSLOduration=1.670725425 podStartE2EDuration="5.022609645s" totalImagesPullingTime="3.35188422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:06.021343338 +0000 UTC m=+72.864506490" watchObservedRunningTime="2026-09-17 17:58:06.022609645 +0000 UTC m=+72.865772812"
Sep 17 17:58:06.547453 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:06.547424    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ssns9_f33d54a6-6442-4f25-84eb-aad9b3c40c0d/serve-healthcheck-canary/0.log"
Sep 17 17:58:06.748431 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:06.748405    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-lwv2h_b6e91b81-c4f3-4032-8fc9-4cdf595b52f3/migrator/0.log"
Sep 17 17:58:06.947165 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:06.947139    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-lwv2h_b6e91b81-c4f3-4032-8fc9-4cdf595b52f3/graceful-termination/0.log"
Sep 17 17:58:07.343828 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:07.343728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:58:07.343828 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:07.343794    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:58:07.344227 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:07.344126    2577 scope.go:117] "RemoveContainer" containerID="1a6cef62dd9a5acc9bf95861e904f0132ec155cebf672af64cdfe612ab162df1"
Sep 17 17:58:07.344289 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:07.344270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cjccs_openshift-console-operator(a8ef87c2-1676-4f2d-ae74-a952d8f18b14)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" podUID="a8ef87c2-1676-4f2d-ae74-a952d8f18b14"
Sep 17 17:58:09.531418 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:09.531371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:09.531859 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:09.531478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:09.531859 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:09.531523    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:09.531859 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:09.531542    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:09.531859 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:09.531614    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:17.531599366 +0000 UTC m=+84.374762516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:09.531859 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:09.531628    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls podName:662b4cbe-fbf3-45ec-a13e-0d0b372d68cf nodeName:}" failed. No retries permitted until 2026-09-17 17:58:17.531621951 +0000 UTC m=+84.374785076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hf8l4" (UID: "662b4cbe-fbf3-45ec-a13e-0d0b372d68cf") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:12.758914 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:12.758880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:58:12.759300 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:12.759030    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:12.759300 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:12.759095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls podName:5e8a92fe-3574-4410-9565-d7926dc55b83 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:28.759079621 +0000 UTC m=+95.602242751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-w49ts" (UID: "5e8a92fe-3574-4410-9565-d7926dc55b83") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:12.859547 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:12.859516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:12.859719 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:12.859577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:12.859719 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:12.859690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle podName:dcd70886-5382-4843-807a-b4c6b7cffa45 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:28.859672214 +0000 UTC m=+95.702835340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle") pod "router-default-5cc8c845db-rbdqs" (UID: "dcd70886-5382-4843-807a-b4c6b7cffa45") : configmap references non-existent config key: service-ca.crt
Sep 17 17:58:12.861702 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:12.861684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd70886-5382-4843-807a-b4c6b7cffa45-metrics-certs\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:17.597191 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:17.597156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:17.597581 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:17.597204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:17.597725 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:17.597706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:17.599495 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:17.599478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/662b4cbe-fbf3-45ec-a13e-0d0b372d68cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hf8l4\" (UID: \"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf\") " pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:17.774103 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:17.774071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hf8l4"
Sep 17 17:58:18.057260 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:18.057228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hf8l4"]
Sep 17 17:58:18.060280 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:18.060254    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662b4cbe_fbf3_45ec_a13e_0d0b372d68cf.slice/crio-5b9b6efcee75dbc4d14c1442cf92027560f32c8f1fcf1bb160a506a656defad3 WatchSource:0}: Error finding container 5b9b6efcee75dbc4d14c1442cf92027560f32c8f1fcf1bb160a506a656defad3: Status 404 returned error can't find the container with id 5b9b6efcee75dbc4d14c1442cf92027560f32c8f1fcf1bb160a506a656defad3
Sep 17 17:58:18.579667 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:18.579635    2577 scope.go:117] "RemoveContainer" containerID="1a6cef62dd9a5acc9bf95861e904f0132ec155cebf672af64cdfe612ab162df1"
Sep 17 17:58:19.013490 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.013454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hf8l4" event={"ID":"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf","Type":"ContainerStarted","Data":"5ef14495da689e35d2f10fac7ab3c25b14c7cab934c291305493e84f42e7237b"}
Sep 17 17:58:19.013921 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.013522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hf8l4" event={"ID":"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf","Type":"ContainerStarted","Data":"82243cda8419463df5c4f7a71dd0791e3e33b3bad77444777d3026f1ff45e0d9"}
Sep 17 17:58:19.013921 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.013553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hf8l4" event={"ID":"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf","Type":"ContainerStarted","Data":"5b9b6efcee75dbc4d14c1442cf92027560f32c8f1fcf1bb160a506a656defad3"}
Sep 17 17:58:19.015271 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.015256    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 17:58:19.015342 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.015306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" event={"ID":"a8ef87c2-1676-4f2d-ae74-a952d8f18b14","Type":"ContainerStarted","Data":"a160f0180b878bdb19fd57f693a6a9d060e8121689e0ead7acb6eee58140f44b"}
Sep 17 17:58:19.015572 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.015544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:58:19.040776 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.040731    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs" podStartSLOduration=19.625152641 podStartE2EDuration="22.040716919s" totalImagesPullingTime="2.415564278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:19.03982517 +0000 UTC m=+85.882988318" watchObservedRunningTime="2026-09-17 17:58:19.040716919 +0000 UTC m=+85.883880067"
Sep 17 17:58:19.143308 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:19.143281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-cjccs"
Sep 17 17:58:20.020547 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:20.020512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hf8l4" event={"ID":"662b4cbe-fbf3-45ec-a13e-0d0b372d68cf","Type":"ContainerStarted","Data":"70726c0a57caa3db6b307c1626df788eec7cab4970627c2d92c1ae0e7eb954d5"}
Sep 17 17:58:20.055009 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:20.054961    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hf8l4" podStartSLOduration=17.282266947 podStartE2EDuration="19.054945966s" totalImagesPullingTime="1.772679019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:20.053217598 +0000 UTC m=+86.896380746" watchObservedRunningTime="2026-09-17 17:58:20.054945966 +0000 UTC m=+86.898109114"
Sep 17 17:58:28.583142 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.583110    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-7bq68"]
Sep 17 17:58:28.585241 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.585223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:28.591669 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.591643    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:58:28.591767 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.591744    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7v2sv\""
Sep 17 17:58:28.593536 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.593521    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:58:28.630613 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.630574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-7bq68"]
Sep 17 17:58:28.682875 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.682844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48g84\" (UniqueName: \"kubernetes.io/projected/01cb9b8e-1c5d-4514-831b-70a38f59b3c4-kube-api-access-48g84\") pod \"downloads-6594f4c75-7bq68\" (UID: \"01cb9b8e-1c5d-4514-831b-70a38f59b3c4\") " pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:28.783411 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.783378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:58:28.783637 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.783458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48g84\" (UniqueName: \"kubernetes.io/projected/01cb9b8e-1c5d-4514-831b-70a38f59b3c4-kube-api-access-48g84\") pod \"downloads-6594f4c75-7bq68\" (UID: \"01cb9b8e-1c5d-4514-831b-70a38f59b3c4\") " pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:28.785844 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.785803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e8a92fe-3574-4410-9565-d7926dc55b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-w49ts\" (UID: \"5e8a92fe-3574-4410-9565-d7926dc55b83\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:58:28.807919 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.807891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48g84\" (UniqueName: \"kubernetes.io/projected/01cb9b8e-1c5d-4514-831b-70a38f59b3c4-kube-api-access-48g84\") pod \"downloads-6594f4c75-7bq68\" (UID: \"01cb9b8e-1c5d-4514-831b-70a38f59b3c4\") " pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:28.884511 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.884412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:28.885058 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.885036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd70886-5382-4843-807a-b4c6b7cffa45-service-ca-bundle\") pod \"router-default-5cc8c845db-rbdqs\" (UID: \"dcd70886-5382-4843-807a-b4c6b7cffa45\") " pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:28.893572 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:28.893548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:29.032675 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:29.032644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"
Sep 17 17:58:29.040607 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:29.040587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:29.200968 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:29.200938    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cb9b8e_1c5d_4514_831b_70a38f59b3c4.slice/crio-cd8b953286f71d15809ac792a6e6470632f12f6e1574efe6f87d6293fc99e058 WatchSource:0}: Error finding container cd8b953286f71d15809ac792a6e6470632f12f6e1574efe6f87d6293fc99e058: Status 404 returned error can't find the container with id cd8b953286f71d15809ac792a6e6470632f12f6e1574efe6f87d6293fc99e058
Sep 17 17:58:29.201918 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:29.201892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-7bq68"]
Sep 17 17:58:29.334426 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:29.334368    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts"]
Sep 17 17:58:29.336741 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:29.336717    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8a92fe_3574_4410_9565_d7926dc55b83.slice/crio-06af96c8d1564b08065b0cd69551a342b19a96d7b8371d6fc5a23b67bc0b7667 WatchSource:0}: Error finding container 06af96c8d1564b08065b0cd69551a342b19a96d7b8371d6fc5a23b67bc0b7667: Status 404 returned error can't find the container with id 06af96c8d1564b08065b0cd69551a342b19a96d7b8371d6fc5a23b67bc0b7667
Sep 17 17:58:29.348174 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:29.348150    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cc8c845db-rbdqs"]
Sep 17 17:58:29.351526 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:29.351491    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd70886_5382_4843_807a_b4c6b7cffa45.slice/crio-6d212b13a397a425e0ad13fccadda010df46d6739b8eb31dd716bdabdd5b58aa WatchSource:0}: Error finding container 6d212b13a397a425e0ad13fccadda010df46d6739b8eb31dd716bdabdd5b58aa: Status 404 returned error can't find the container with id 6d212b13a397a425e0ad13fccadda010df46d6739b8eb31dd716bdabdd5b58aa
Sep 17 17:58:30.055253 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:30.054964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cc8c845db-rbdqs" event={"ID":"dcd70886-5382-4843-807a-b4c6b7cffa45","Type":"ContainerStarted","Data":"64b0122f40a0a2727144f7492181ab731349bb7ba9181127d331cc63fbf9723d"}
Sep 17 17:58:30.055253 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:30.055019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cc8c845db-rbdqs" event={"ID":"dcd70886-5382-4843-807a-b4c6b7cffa45","Type":"ContainerStarted","Data":"6d212b13a397a425e0ad13fccadda010df46d6739b8eb31dd716bdabdd5b58aa"}
Sep 17 17:58:30.056760 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:30.056712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts" event={"ID":"5e8a92fe-3574-4410-9565-d7926dc55b83","Type":"ContainerStarted","Data":"06af96c8d1564b08065b0cd69551a342b19a96d7b8371d6fc5a23b67bc0b7667"}
Sep 17 17:58:30.058117 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:30.058086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-7bq68" event={"ID":"01cb9b8e-1c5d-4514-831b-70a38f59b3c4","Type":"ContainerStarted","Data":"cd8b953286f71d15809ac792a6e6470632f12f6e1574efe6f87d6293fc99e058"}
Sep 17 17:58:30.089141 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:30.089072    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5cc8c845db-rbdqs" podStartSLOduration=34.089051987 podStartE2EDuration="34.089051987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:30.088539176 +0000 UTC m=+96.931702328" watchObservedRunningTime="2026-09-17 17:58:30.089051987 +0000 UTC m=+96.932215137"
Sep 17 17:58:31.041737 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.041698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:31.046404 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.046345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:31.065708 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.064463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:31.067592 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.067546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cc8c845db-rbdqs"
Sep 17 17:58:31.820431 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.820397    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"]
Sep 17 17:58:31.822606 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.822588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:31.825390 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.825369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:58:31.825500 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.825404    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cqhtj\""
Sep 17 17:58:31.838325 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.838298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"]
Sep 17 17:58:31.911118 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:31.911081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07a0b8c2-c49e-4531-b9b6-794a59f0452f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-px4gp\" (UID: \"07a0b8c2-c49e-4531-b9b6-794a59f0452f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:32.012026 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.011995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07a0b8c2-c49e-4531-b9b6-794a59f0452f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-px4gp\" (UID: \"07a0b8c2-c49e-4531-b9b6-794a59f0452f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:32.012194 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:32.012165    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 17:58:32.012275 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:32.012243    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0b8c2-c49e-4531-b9b6-794a59f0452f-tls-certificates podName:07a0b8c2-c49e-4531-b9b6-794a59f0452f nodeName:}" failed. No retries permitted until 2026-09-17 17:58:32.512222086 +0000 UTC m=+99.355385237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/07a0b8c2-c49e-4531-b9b6-794a59f0452f-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-px4gp" (UID: "07a0b8c2-c49e-4531-b9b6-794a59f0452f") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 17:58:32.068243 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.068210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts" event={"ID":"5e8a92fe-3574-4410-9565-d7926dc55b83","Type":"ContainerStarted","Data":"8befb94b44b6137a20f2a6e1d71bfb75c0135a104797609ede3908fd267a4ebd"}
Sep 17 17:58:32.091149 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.091055    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-w49ts" podStartSLOduration=34.225570816 podStartE2EDuration="36.091035078s" totalImagesPullingTime="1.865464262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:32.090629408 +0000 UTC m=+98.933792783" watchObservedRunningTime="2026-09-17 17:58:32.091035078 +0000 UTC m=+98.934198229"
Sep 17 17:58:32.516657 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.516623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07a0b8c2-c49e-4531-b9b6-794a59f0452f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-px4gp\" (UID: \"07a0b8c2-c49e-4531-b9b6-794a59f0452f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:32.519313 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.519280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07a0b8c2-c49e-4531-b9b6-794a59f0452f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-px4gp\" (UID: \"07a0b8c2-c49e-4531-b9b6-794a59f0452f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:32.571893 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.571857    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbbc46cc7-rzbjp"]
Sep 17 17:58:32.591152 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.591124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.598300 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.598273    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:58:32.599504 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.599295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:58:32.599730 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.599709    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:58:32.600093 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.600074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h8dtw\""
Sep 17 17:58:32.600187 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.600111    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:58:32.600187 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.600140    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:58:32.611661 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.611633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbbc46cc7-rzbjp"]
Sep 17 17:58:32.719104 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.719064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-oauth-serving-cert\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.719288 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.719119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-serving-cert\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.719288 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.719166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-oauth-config\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.719288 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.719207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-config\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.719288 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.719263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-service-ca\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.719508 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.719306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpfd\" (UniqueName: \"kubernetes.io/projected/028158c0-fe2c-4a67-9a58-e861cfceae7b-kube-api-access-ptpfd\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.732985 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.732953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:32.820277 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.820238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-serving-cert\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.820426 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.820316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-oauth-config\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.820426 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.820371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-config\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.820426 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.820403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-service-ca\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.820610 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.820431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpfd\" (UniqueName: \"kubernetes.io/projected/028158c0-fe2c-4a67-9a58-e861cfceae7b-kube-api-access-ptpfd\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.820610 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.820509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-oauth-serving-cert\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.821468 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.821393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-oauth-serving-cert\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.822135 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.822087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-service-ca\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.822135 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.822095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-config\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.829239 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.824801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-oauth-config\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.829239 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.826313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-serving-cert\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.832388 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.832359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpfd\" (UniqueName: \"kubernetes.io/projected/028158c0-fe2c-4a67-9a58-e861cfceae7b-kube-api-access-ptpfd\") pod \"console-bbbc46cc7-rzbjp\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") " pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:32.902878 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:32.902846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:33.033107 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:33.033073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"]
Sep 17 17:58:33.036827 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:33.036799    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a0b8c2_c49e_4531_b9b6_794a59f0452f.slice/crio-9460c629975fb1e87d412df5823aa046cf65dc19fd5416250d35d0e666fddb8a WatchSource:0}: Error finding container 9460c629975fb1e87d412df5823aa046cf65dc19fd5416250d35d0e666fddb8a: Status 404 returned error can't find the container with id 9460c629975fb1e87d412df5823aa046cf65dc19fd5416250d35d0e666fddb8a
Sep 17 17:58:33.072708 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:33.072622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp" event={"ID":"07a0b8c2-c49e-4531-b9b6-794a59f0452f","Type":"ContainerStarted","Data":"9460c629975fb1e87d412df5823aa046cf65dc19fd5416250d35d0e666fddb8a"}
Sep 17 17:58:33.201673 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:33.201636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbbc46cc7-rzbjp"]
Sep 17 17:58:33.204646 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:33.204613    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028158c0_fe2c_4a67_9a58_e861cfceae7b.slice/crio-eea9b54dee666c208bef376000345efcf9d6981b0ee56ea13fcf4648869144a4 WatchSource:0}: Error finding container eea9b54dee666c208bef376000345efcf9d6981b0ee56ea13fcf4648869144a4: Status 404 returned error can't find the container with id eea9b54dee666c208bef376000345efcf9d6981b0ee56ea13fcf4648869144a4
Sep 17 17:58:33.931538 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:33.931496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:58:33.936889 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:33.936846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ceb131a8-2b87-49d7-855e-2b2b89c9fa99-metrics-tls\") pod \"dns-default-s5qlj\" (UID: \"ceb131a8-2b87-49d7-855e-2b2b89c9fa99\") " pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:58:34.082543 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:34.082506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbbc46cc7-rzbjp" event={"ID":"028158c0-fe2c-4a67-9a58-e861cfceae7b","Type":"ContainerStarted","Data":"eea9b54dee666c208bef376000345efcf9d6981b0ee56ea13fcf4648869144a4"}
Sep 17 17:58:34.194648 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:34.194556    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zr8ck\""
Sep 17 17:58:34.200017 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:34.199988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:58:34.993212 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:34.993126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s5qlj"]
Sep 17 17:58:34.998652 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:34.998614    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb131a8_2b87_49d7_855e_2b2b89c9fa99.slice/crio-226b7bff15242e9aabc6a084941f74294e2171b5137e813aa30457b4a4be9008 WatchSource:0}: Error finding container 226b7bff15242e9aabc6a084941f74294e2171b5137e813aa30457b4a4be9008: Status 404 returned error can't find the container with id 226b7bff15242e9aabc6a084941f74294e2171b5137e813aa30457b4a4be9008
Sep 17 17:58:35.088296 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.088247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s5qlj" event={"ID":"ceb131a8-2b87-49d7-855e-2b2b89c9fa99","Type":"ContainerStarted","Data":"226b7bff15242e9aabc6a084941f74294e2171b5137e813aa30457b4a4be9008"}
Sep 17 17:58:35.090317 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.090290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp" event={"ID":"07a0b8c2-c49e-4531-b9b6-794a59f0452f","Type":"ContainerStarted","Data":"e38a96deb444e3ace1c81068c2eb60331d624f0bfd334caaf601a1f4121abdcb"}
Sep 17 17:58:35.090709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.090606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:35.097065 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.097040    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp"
Sep 17 17:58:35.115099 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.114967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-px4gp" podStartSLOduration=2.222730548 podStartE2EDuration="4.114948247s" totalImagesPullingTime="1.892217699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:35.114876856 +0000 UTC m=+101.958040006" watchObservedRunningTime="2026-09-17 17:58:35.114948247 +0000 UTC m=+101.958111402"
Sep 17 17:58:35.897124 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.897091    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"]
Sep 17 17:58:35.938772 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.938726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"]
Sep 17 17:58:35.938941 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.938858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:35.942314 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.942284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:58:35.942447 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.942373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:58:35.942447 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.942374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rht2j\""
Sep 17 17:58:35.944638 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:35.944504    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:58:36.050867 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.050820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.051090 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.050912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.051090 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.050935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.051090 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.050952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks77w\" (UniqueName: \"kubernetes.io/projected/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-kube-api-access-ks77w\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.151943 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.151900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.152374 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.152004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.152374 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.152040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.152374 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:36.152053    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 17:58:36.152374 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.152066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks77w\" (UniqueName: \"kubernetes.io/projected/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-kube-api-access-ks77w\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.152374 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:58:36.152120    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-tls podName:1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:36.65210243 +0000 UTC m=+103.495265580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-c6w6x" (UID: "1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626") : secret "prometheus-operator-tls" not found
Sep 17 17:58:36.152945 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.152916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.155163 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.155118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.162088 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.162061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks77w\" (UniqueName: \"kubernetes.io/projected/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-kube-api-access-ks77w\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.656878 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.656837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.660321 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.660288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c6w6x\" (UID: \"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.850224 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.850140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"
Sep 17 17:58:36.977898 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:36.977857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b9jfv"
Sep 17 17:58:38.742304 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:38.742194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x"]
Sep 17 17:58:39.027103 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:39.027027    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e70e7bd_5fdc_44a9_9ffb_ba86c07fe626.slice/crio-04412b7eef9909df258a9927c26635c3f5f84863793e9e97ef989de93302a6ee WatchSource:0}: Error finding container 04412b7eef9909df258a9927c26635c3f5f84863793e9e97ef989de93302a6ee: Status 404 returned error can't find the container with id 04412b7eef9909df258a9927c26635c3f5f84863793e9e97ef989de93302a6ee
Sep 17 17:58:39.109030 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:39.108991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x" event={"ID":"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626","Type":"ContainerStarted","Data":"04412b7eef9909df258a9927c26635c3f5f84863793e9e97ef989de93302a6ee"}
Sep 17 17:58:40.115312 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:40.115227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s5qlj" event={"ID":"ceb131a8-2b87-49d7-855e-2b2b89c9fa99","Type":"ContainerStarted","Data":"bd4d518bb20af3a717d2b7684ab904de308ec71aec934e89711bc20ff3f27b98"}
Sep 17 17:58:40.115312 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:40.115288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s5qlj" event={"ID":"ceb131a8-2b87-49d7-855e-2b2b89c9fa99","Type":"ContainerStarted","Data":"e08b31dc148cf82c31bba99124cc404ff69ecf128a18813e12e8ab5548783e0b"}
Sep 17 17:58:40.115879 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:40.115321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:58:40.117520 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:40.117157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbbc46cc7-rzbjp" event={"ID":"028158c0-fe2c-4a67-9a58-e861cfceae7b","Type":"ContainerStarted","Data":"a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d"}
Sep 17 17:58:40.186596 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:40.186454    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bbbc46cc7-rzbjp" podStartSLOduration=2.328757841 podStartE2EDuration="8.186433793s" totalImagesPullingTime="5.857675952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:40.186174651 +0000 UTC m=+107.029337800" watchObservedRunningTime="2026-09-17 17:58:40.186433793 +0000 UTC m=+107.029596938"
Sep 17 17:58:40.187601 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:40.187535    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s5qlj" podStartSLOduration=67.135691752 podStartE2EDuration="1m11.187523534s" totalImagesPullingTime="4.051831782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:40.151310129 +0000 UTC m=+106.994473276" watchObservedRunningTime="2026-09-17 17:58:40.187523534 +0000 UTC m=+107.030686684"
Sep 17 17:58:42.903535 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:42.903502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:42.904061 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:42.903582    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:42.908629 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:42.908608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:43.133413 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:43.133378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:58:47.145305 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.145207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-7bq68" event={"ID":"01cb9b8e-1c5d-4514-831b-70a38f59b3c4","Type":"ContainerStarted","Data":"6eea928788eb712049884e4c62aec210f4e418707e6c251dbd24e0d838b8ee31"}
Sep 17 17:58:47.145774 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.145448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:47.147815 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.147738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x" event={"ID":"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626","Type":"ContainerStarted","Data":"237d652a5fac25a55fb0fc712c6f8b6d10ee74c7f835e739a97539385ce3db00"}
Sep 17 17:58:47.147815 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.147788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x" event={"ID":"1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626","Type":"ContainerStarted","Data":"1c923a4936375ccd1a547e7706e1420dcda54774a24124fd135df5dfe93824ea"}
Sep 17 17:58:47.157311 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.157283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-7bq68"
Sep 17 17:58:47.292261 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.292196    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-7bq68" podStartSLOduration=1.6392190869999999 podStartE2EDuration="19.292175785s" totalImagesPullingTime="17.652956698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:47.23642664 +0000 UTC m=+114.079589803" watchObservedRunningTime="2026-09-17 17:58:47.292175785 +0000 UTC m=+114.135338932"
Sep 17 17:58:47.642360 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.642303    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c6w6x" podStartSLOduration=4.871663288 podStartE2EDuration="12.642271535s" totalImagesPullingTime="7.770608247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:47.356697179 +0000 UTC m=+114.199860328" watchObservedRunningTime="2026-09-17 17:58:47.642271535 +0000 UTC m=+114.485434682"
Sep 17 17:58:47.643654 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:47.643594    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbbc46cc7-rzbjp"]
Sep 17 17:58:49.659262 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.659224    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"]
Sep 17 17:58:49.683843 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.683809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.691617 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.691379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:58:49.692554 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.692525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:58:49.692679 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.692525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jcdhv\""
Sep 17 17:58:49.702203 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.702157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"]
Sep 17 17:58:49.760739 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.760704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nl5km"]
Sep 17 17:58:49.776149 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.776125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.781658 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.781630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwrpp\""
Sep 17 17:58:49.781828 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.781633    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:58:49.782036 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.782017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:58:49.782695 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.782678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:58:49.871441 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b654e052-4eaa-45ab-be80-4bda1f808076-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.871709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61556aed-1c07-4f91-a91d-3ead9d411694-metrics-client-ca\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.871709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-textfile\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.871709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-wtmp\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.871709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-sys\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.871709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-accelerators-collector-config\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.871709 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9hx\" (UniqueName: \"kubernetes.io/projected/61556aed-1c07-4f91-a91d-3ead9d411694-kube-api-access-cn9hx\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.872034 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b654e052-4eaa-45ab-be80-4bda1f808076-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.872034 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.872034 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-root\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.872034 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.871966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q945t\" (UniqueName: \"kubernetes.io/projected/b654e052-4eaa-45ab-be80-4bda1f808076-kube-api-access-q945t\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.872034 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.872014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-tls\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.872270 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.872044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b654e052-4eaa-45ab-be80-4bda1f808076-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.973166 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-textfile\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973166 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-wtmp\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973442 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-sys\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973442 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-wtmp\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973442 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-accelerators-collector-config\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973442 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-sys\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973442 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9hx\" (UniqueName: \"kubernetes.io/projected/61556aed-1c07-4f91-a91d-3ead9d411694-kube-api-access-cn9hx\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973442 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-textfile\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973768 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b654e052-4eaa-45ab-be80-4bda1f808076-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.973768 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973768 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-root\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973768 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q945t\" (UniqueName: \"kubernetes.io/projected/b654e052-4eaa-45ab-be80-4bda1f808076-kube-api-access-q945t\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.973768 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-tls\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.973768 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b654e052-4eaa-45ab-be80-4bda1f808076-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.974054 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b654e052-4eaa-45ab-be80-4bda1f808076-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.974054 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61556aed-1c07-4f91-a91d-3ead9d411694-metrics-client-ca\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.974054 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-accelerators-collector-config\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.974054 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.973983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61556aed-1c07-4f91-a91d-3ead9d411694-root\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.974497 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.974473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61556aed-1c07-4f91-a91d-3ead9d411694-metrics-client-ca\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.975082 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.975052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b654e052-4eaa-45ab-be80-4bda1f808076-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.976478 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.976433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.976728 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.976678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b654e052-4eaa-45ab-be80-4bda1f808076-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.977080 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.977055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b654e052-4eaa-45ab-be80-4bda1f808076-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:49.977493 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.977476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61556aed-1c07-4f91-a91d-3ead9d411694-node-exporter-tls\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.993354 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.993319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9hx\" (UniqueName: \"kubernetes.io/projected/61556aed-1c07-4f91-a91d-3ead9d411694-kube-api-access-cn9hx\") pod \"node-exporter-nl5km\" (UID: \"61556aed-1c07-4f91-a91d-3ead9d411694\") " pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:49.995787 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:49.995758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q945t\" (UniqueName: \"kubernetes.io/projected/b654e052-4eaa-45ab-be80-4bda1f808076-kube-api-access-q945t\") pod \"openshift-state-metrics-86c5bc79d6-2kszl\" (UID: \"b654e052-4eaa-45ab-be80-4bda1f808076\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:50.087035 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.086996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nl5km"
Sep 17 17:58:50.113264 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:50.113235    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61556aed_1c07_4f91_a91d_3ead9d411694.slice/crio-40dc2e2d79e7e31334d2443136896d3faaee04020105d848fad87ba14b99955b WatchSource:0}: Error finding container 40dc2e2d79e7e31334d2443136896d3faaee04020105d848fad87ba14b99955b: Status 404 returned error can't find the container with id 40dc2e2d79e7e31334d2443136896d3faaee04020105d848fad87ba14b99955b
Sep 17 17:58:50.123837 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.123807    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s5qlj"
Sep 17 17:58:50.162376 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.162337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nl5km" event={"ID":"61556aed-1c07-4f91-a91d-3ead9d411694","Type":"ContainerStarted","Data":"40dc2e2d79e7e31334d2443136896d3faaee04020105d848fad87ba14b99955b"}
Sep 17 17:58:50.295287 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.295211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"
Sep 17 17:58:50.627090 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.626145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl"]
Sep 17 17:58:50.970478 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.970446    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:50.998579 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:50.998545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.003596 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.003548    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:58:51.003722 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.003664    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:58:51.004218 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004197    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:58:51.004326 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:58:51.004387 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:58:51.004459 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:58:51.004518 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r9sgp\""
Sep 17 17:58:51.004649 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004628    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:58:51.004649 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.004645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:58:51.005019 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.005001    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:58:51.016777 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.016751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:51.171531 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.171401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl" event={"ID":"b654e052-4eaa-45ab-be80-4bda1f808076","Type":"ContainerStarted","Data":"155c243c6450d11cf141421cdb9aa41b1f0c6f2a5ad5c4cd361e664b0fa1e81c"}
Sep 17 17:58:51.171531 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.171465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl" event={"ID":"b654e052-4eaa-45ab-be80-4bda1f808076","Type":"ContainerStarted","Data":"b3d8dd74112c0637615347c4d0d4a2f26eb21c5a9c3d02a9467ae35808211487"}
Sep 17 17:58:51.171531 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.171492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl" event={"ID":"b654e052-4eaa-45ab-be80-4bda1f808076","Type":"ContainerStarted","Data":"01a975f0f1dbd84f99f447d16ed135c4b9e3586adfeba065a65047522da4b510"}
Sep 17 17:58:51.186338 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186338 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186551 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49f8\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-kube-api-access-t49f8\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186551 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-web-config\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186551 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186551 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186551 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186551 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186926 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186926 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186926 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-out\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186926 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.186926 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.186733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-out\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49f8\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-kube-api-access-t49f8\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-web-config\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.287583 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.288096 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.288096 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.288096 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.288096 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.288096 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.287728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.291579 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.288655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.291579 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.288993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.291579 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.289647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.296485 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.296460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.297086 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.297066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.297593 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.297528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.297884 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.297865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.299497 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.299457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-web-config\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.299900 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.299858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-out\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.300405 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.300368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.301124 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.301103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49f8\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-kube-api-access-t49f8\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.303177 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.303154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.304772 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.304744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.310394 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.310366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:51.755091 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:51.755063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:51.775622 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:51.775273    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c758a6_22ff_47d4_a78f_039e83ef13a6.slice/crio-b4546e30ac9cebeee2df98e058543e8a2ba3aab802e086377a85b51b49fc4dcd WatchSource:0}: Error finding container b4546e30ac9cebeee2df98e058543e8a2ba3aab802e086377a85b51b49fc4dcd: Status 404 returned error can't find the container with id b4546e30ac9cebeee2df98e058543e8a2ba3aab802e086377a85b51b49fc4dcd
Sep 17 17:58:52.178973 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:52.178795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nl5km" event={"ID":"61556aed-1c07-4f91-a91d-3ead9d411694","Type":"ContainerStarted","Data":"ed3117d4ed085d7b2dbecc8d70ad5b999a7fcc4caf80cbc54b0f7dd1f0630d6f"}
Sep 17 17:58:52.181103 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:52.181047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"b4546e30ac9cebeee2df98e058543e8a2ba3aab802e086377a85b51b49fc4dcd"}
Sep 17 17:58:53.189087 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.188938    2577 generic.go:356] "Generic (PLEG): container finished" podID="61556aed-1c07-4f91-a91d-3ead9d411694" containerID="ed3117d4ed085d7b2dbecc8d70ad5b999a7fcc4caf80cbc54b0f7dd1f0630d6f" exitCode=0
Sep 17 17:58:53.189087 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.189045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nl5km" event={"ID":"61556aed-1c07-4f91-a91d-3ead9d411694","Type":"ContainerDied","Data":"ed3117d4ed085d7b2dbecc8d70ad5b999a7fcc4caf80cbc54b0f7dd1f0630d6f"}
Sep 17 17:58:53.194846 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.194355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl" event={"ID":"b654e052-4eaa-45ab-be80-4bda1f808076","Type":"ContainerStarted","Data":"fdab3273988624431f9bb31a971577cc8df90bef4d9e1e482ff2509b9c1d9790"}
Sep 17 17:58:53.947982 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.947927    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2kszl" podStartSLOduration=3.132119813 podStartE2EDuration="4.947903879s" totalImagesPullingTime="1.815784066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:53.259742648 +0000 UTC m=+120.102905797" watchObservedRunningTime="2026-09-17 17:58:53.947903879 +0000 UTC m=+120.791067027"
Sep 17 17:58:53.948879 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.948856    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"]
Sep 17 17:58:53.971201 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.971177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:53.978701 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.978678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:58:53.978960 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.978940    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9ff815shiljc0\""
Sep 17 17:58:53.979206 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.979183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9vlnh\""
Sep 17 17:58:53.979582 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.979529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"]
Sep 17 17:58:53.979702 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.979686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:58:53.980064 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.980034    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:58:53.980362 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.980339    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:58:53.980590 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:53.980544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:58:54.113740 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57d081a3-b6f5-4cd1-aa09-44eeb327809f-metrics-client-ca\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.113903 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.113903 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-tls\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.113903 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.113903 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.113903 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-grpc-tls\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.114150 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.114150 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.113977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqdc\" (UniqueName: \"kubernetes.io/projected/57d081a3-b6f5-4cd1-aa09-44eeb327809f-kube-api-access-vvqdc\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.201254 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.201160    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad" exitCode=0
Sep 17 17:58:54.201796 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.201458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"}
Sep 17 17:58:54.202584 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.202436    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"]
Sep 17 17:58:54.215060 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-tls\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215195 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215195 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215195 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-grpc-tls\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215195 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215401 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqdc\" (UniqueName: \"kubernetes.io/projected/57d081a3-b6f5-4cd1-aa09-44eeb327809f-kube-api-access-vvqdc\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215401 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57d081a3-b6f5-4cd1-aa09-44eeb327809f-metrics-client-ca\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.215401 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.215290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.216455 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.216400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57d081a3-b6f5-4cd1-aa09-44eeb327809f-metrics-client-ca\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.218537 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.218186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.218784 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.218713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-tls\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.219839 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.219792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.220317 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.220037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.220317 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.220272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.220469 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.220332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57d081a3-b6f5-4cd1-aa09-44eeb327809f-secret-grpc-tls\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.223070 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.223030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nl5km" event={"ID":"61556aed-1c07-4f91-a91d-3ead9d411694","Type":"ContainerStarted","Data":"34da6b39f6a41328a8c1a7061e42cbb37211566fe4672c6c5bdc7d51016a332f"}
Sep 17 17:58:54.223178 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.223082    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d89c9995d-q2j5p"]
Sep 17 17:58:54.223178 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.223152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.226355 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.226335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lk5qw\""
Sep 17 17:58:54.230818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.230739    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:58:54.233712 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.231937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqdc\" (UniqueName: \"kubernetes.io/projected/57d081a3-b6f5-4cd1-aa09-44eeb327809f-kube-api-access-vvqdc\") pod \"thanos-querier-5bf5dbb977-nbqmw\" (UID: \"57d081a3-b6f5-4cd1-aa09-44eeb327809f\") " pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.248522 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.248487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nl5km" event={"ID":"61556aed-1c07-4f91-a91d-3ead9d411694","Type":"ContainerStarted","Data":"c252c1f98d5e920f86d37280fbbfc3f9b54905731aad42cb5a529e17466964e7"}
Sep 17 17:58:54.248677 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.248604    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"]
Sep 17 17:58:54.248677 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.248638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.248794 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.248643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d89c9995d-q2j5p"]
Sep 17 17:58:54.257126 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.257102    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:58:54.292250 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.292189    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nl5km" podStartSLOduration=3.513445529 podStartE2EDuration="5.292164916s" totalImagesPullingTime="1.69025218s" totalInitContainerRuntime="88.467207ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:54.290796433 +0000 UTC m=+121.133959583" watchObservedRunningTime="2026-09-17 17:58:54.292164916 +0000 UTC m=+121.135328065"
Sep 17 17:58:54.294595 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.294460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:58:54.418307 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz4v\" (UniqueName: \"kubernetes.io/projected/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-kube-api-access-pgz4v\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.418307 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3d736b9-6be9-4d09-ab3f-2c8d307aa98c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-8n59t\" (UID: \"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.418307 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7hs\" (UniqueName: \"kubernetes.io/projected/f3d736b9-6be9-4d09-ab3f-2c8d307aa98c-kube-api-access-5p7hs\") pod \"monitoring-plugin-5984459ff8-8n59t\" (UID: \"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.418307 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-config\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.418986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-oauth-config\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.418986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-trusted-ca-bundle\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.418986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-serving-cert\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.418986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-oauth-serving-cert\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.418986 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.418555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-service-ca\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.519779 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.519694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-config\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.519779 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.519751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-oauth-config\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.519999 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.519782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-trusted-ca-bundle\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.519999 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.519807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-serving-cert\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.519999 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.519836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-oauth-serving-cert\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.519999 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.519890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-service-ca\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.520206 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.520109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz4v\" (UniqueName: \"kubernetes.io/projected/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-kube-api-access-pgz4v\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.520206 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.520154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3d736b9-6be9-4d09-ab3f-2c8d307aa98c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-8n59t\" (UID: \"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.520206 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.520186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7hs\" (UniqueName: \"kubernetes.io/projected/f3d736b9-6be9-4d09-ab3f-2c8d307aa98c-kube-api-access-5p7hs\") pod \"monitoring-plugin-5984459ff8-8n59t\" (UID: \"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.521088 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.520518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-config\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.521088 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.520732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-trusted-ca-bundle\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.521088 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.521000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-service-ca\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.521301 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.521185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-oauth-serving-cert\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.523172 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.523126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-oauth-config\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.523337 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.523317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3d736b9-6be9-4d09-ab3f-2c8d307aa98c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-8n59t\" (UID: \"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.524038 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.524014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-serving-cert\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.531440 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.531419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7hs\" (UniqueName: \"kubernetes.io/projected/f3d736b9-6be9-4d09-ab3f-2c8d307aa98c-kube-api-access-5p7hs\") pod \"monitoring-plugin-5984459ff8-8n59t\" (UID: \"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.531981 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.531955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz4v\" (UniqueName: \"kubernetes.io/projected/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-kube-api-access-pgz4v\") pod \"console-6d89c9995d-q2j5p\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") " pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.536687 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.536664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:54.563518 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.563491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:58:54.621406 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.620751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"]
Sep 17 17:58:54.855998 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.855924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"]
Sep 17 17:58:54.861753 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:54.861717    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d736b9_6be9_4d09_ab3f_2c8d307aa98c.slice/crio-33d5440b1d27c8a71188ed6241285a2f6fff724f4c6cdb19991df4300a7381d0 WatchSource:0}: Error finding container 33d5440b1d27c8a71188ed6241285a2f6fff724f4c6cdb19991df4300a7381d0: Status 404 returned error can't find the container with id 33d5440b1d27c8a71188ed6241285a2f6fff724f4c6cdb19991df4300a7381d0
Sep 17 17:58:54.895386 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:54.895314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d89c9995d-q2j5p"]
Sep 17 17:58:54.898023 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:54.897991    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1974a01e_8c72_4dbf_a9c8_7944f2bf54e1.slice/crio-135f4a2022fecc6c5f64806000714e27a5c8cd5e6ff277ec9b10e85f25b810be WatchSource:0}: Error finding container 135f4a2022fecc6c5f64806000714e27a5c8cd5e6ff277ec9b10e85f25b810be: Status 404 returned error can't find the container with id 135f4a2022fecc6c5f64806000714e27a5c8cd5e6ff277ec9b10e85f25b810be
Sep 17 17:58:55.214386 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:55.214342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t" event={"ID":"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c","Type":"ContainerStarted","Data":"33d5440b1d27c8a71188ed6241285a2f6fff724f4c6cdb19991df4300a7381d0"}
Sep 17 17:58:55.216525 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:55.216492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"cf224e1c7a5f79b2b7ab6601b9633d3289950faaa2090e4f4eb36eb3d87c3fc9"}
Sep 17 17:58:55.218886 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:55.218861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89c9995d-q2j5p" event={"ID":"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1","Type":"ContainerStarted","Data":"382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f"}
Sep 17 17:58:55.219172 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:55.218905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89c9995d-q2j5p" event={"ID":"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1","Type":"ContainerStarted","Data":"135f4a2022fecc6c5f64806000714e27a5c8cd5e6ff277ec9b10e85f25b810be"}
Sep 17 17:58:55.253906 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:55.253695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d89c9995d-q2j5p" podStartSLOduration=1.253674731 podStartE2EDuration="1.253674731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:55.252326185 +0000 UTC m=+122.095489336" watchObservedRunningTime="2026-09-17 17:58:55.253674731 +0000 UTC m=+122.096837884"
Sep 17 17:58:58.233336 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.233304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"49882acfb90034f50b4a40c3ca21c123f015c00c059f44477f4eb238d71160e2"}
Sep 17 17:58:58.233699 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.233392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"b351ce94349b5f023669c638fcce665461d49ce08506d4e60eb5be41a3c617e1"}
Sep 17 17:58:58.233699 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.233433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"6145987d2bbbcc915cbff26c2caec371dfd735749e56a8409a2019e504cc8c34"}
Sep 17 17:58:58.236066 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.236044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"}
Sep 17 17:58:58.236159 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.236101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"}
Sep 17 17:58:58.236159 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.236139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"}
Sep 17 17:58:58.237413 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.237392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t" event={"ID":"f3d736b9-6be9-4d09-ab3f-2c8d307aa98c","Type":"ContainerStarted","Data":"4316d28184cc3f4f9f6204dc46c8a532d8c3907843378ec815e8c7acfe72c24b"}
Sep 17 17:58:58.237614 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.237597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:58.242681 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.242663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t"
Sep 17 17:58:58.269967 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.269919    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-8n59t" podStartSLOduration=1.320643547 podStartE2EDuration="4.269904318s" totalImagesPullingTime="2.949260771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:58.269235774 +0000 UTC m=+125.112398923" watchObservedRunningTime="2026-09-17 17:58:58.269904318 +0000 UTC m=+125.113067515"
Sep 17 17:58:58.843176 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.843144    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d89c9995d-q2j5p"]
Sep 17 17:58:58.944871 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.944834    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d568fc8d-dn4s6"]
Sep 17 17:58:58.966077 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.966042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:58.979714 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:58.979131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d568fc8d-dn4s6"]
Sep 17 17:58:59.077993 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.077953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-config\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.078157 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.078005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-service-ca\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.078157 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.078098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-serving-cert\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.078157 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.078147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-oauth-serving-cert\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.078293 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.078190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-oauth-config\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.078293 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.078236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44rf\" (UniqueName: \"kubernetes.io/projected/86166a37-8a0e-40eb-8cbf-c21386e76d6e-kube-api-access-s44rf\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.078360 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.078328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-trusted-ca-bundle\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.178834 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.178798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-service-ca\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.178845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-serving-cert\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.178866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-oauth-serving-cert\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.178899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-oauth-config\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.178941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44rf\" (UniqueName: \"kubernetes.io/projected/86166a37-8a0e-40eb-8cbf-c21386e76d6e-kube-api-access-s44rf\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.178977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-trusted-ca-bundle\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179278 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.179027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-config\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179715 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.179684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-service-ca\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179846 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.179686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-oauth-serving-cert\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.179846 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.179788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-config\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.180161 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.180136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-trusted-ca-bundle\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.181376 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.181353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-serving-cert\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.181376 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.181363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-oauth-config\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.214055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.214023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44rf\" (UniqueName: \"kubernetes.io/projected/86166a37-8a0e-40eb-8cbf-c21386e76d6e-kube-api-access-s44rf\") pod \"console-58d568fc8d-dn4s6\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") " pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.245351 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.245308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"}
Sep 17 17:58:59.245724 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.245381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"}
Sep 17 17:58:59.281369 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.281335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:58:59.773344 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:58:59.773316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d568fc8d-dn4s6"]
Sep 17 17:58:59.777764 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:58:59.777740    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86166a37_8a0e_40eb_8cbf_c21386e76d6e.slice/crio-7c9d3c888a2347942e1b7c7b391bb86561020bb0b4c90f179a06ac40bf08b023 WatchSource:0}: Error finding container 7c9d3c888a2347942e1b7c7b391bb86561020bb0b4c90f179a06ac40bf08b023: Status 404 returned error can't find the container with id 7c9d3c888a2347942e1b7c7b391bb86561020bb0b4c90f179a06ac40bf08b023
Sep 17 17:59:00.253592 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.253544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"656f964c3a3fac4c83f6e8c983beda8514c8f2656cf564cccecc37e90f5ec15c"}
Sep 17 17:59:00.254108 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.253631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"a754554b2d5090df7a19295f6818e0626cae303009c349a31b8e485ce96a3353"}
Sep 17 17:59:00.254108 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.253672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" event={"ID":"57d081a3-b6f5-4cd1-aa09-44eeb327809f","Type":"ContainerStarted","Data":"c37e4a825172b04163d2fe9418576ea5ac0c9a99f4282e0e8ad2063ae4270fd2"}
Sep 17 17:59:00.254108 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.253718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:59:00.256914 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.256886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerStarted","Data":"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"}
Sep 17 17:59:00.258615 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.258588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d568fc8d-dn4s6" event={"ID":"86166a37-8a0e-40eb-8cbf-c21386e76d6e","Type":"ContainerStarted","Data":"f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5"}
Sep 17 17:59:00.258733 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.258628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d568fc8d-dn4s6" event={"ID":"86166a37-8a0e-40eb-8cbf-c21386e76d6e","Type":"ContainerStarted","Data":"7c9d3c888a2347942e1b7c7b391bb86561020bb0b4c90f179a06ac40bf08b023"}
Sep 17 17:59:00.284893 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.284847    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw" podStartSLOduration=2.821734206 podStartE2EDuration="7.28483324s" totalImagesPullingTime="4.463099034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:00.283656552 +0000 UTC m=+127.126819696" watchObservedRunningTime="2026-09-17 17:59:00.28483324 +0000 UTC m=+127.127996387"
Sep 17 17:59:00.307631 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.307580    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58d568fc8d-dn4s6" podStartSLOduration=2.307552436 podStartE2EDuration="2.307552436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:00.305676252 +0000 UTC m=+127.148839400" watchObservedRunningTime="2026-09-17 17:59:00.307552436 +0000 UTC m=+127.150715583"
Sep 17 17:59:00.349773 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:00.349724    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.5203216250000002 podStartE2EDuration="10.349710093s" totalImagesPullingTime="6.808565509s" totalInitContainerRuntime="20.822959ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:00.349064167 +0000 UTC m=+127.192227315" watchObservedRunningTime="2026-09-17 17:59:00.349710093 +0000 UTC m=+127.192873241"
Sep 17 17:59:04.327178 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:04.327141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:59:04.329452 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:04.329424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761dc4fd-5005-48a1-9874-bfa11455f308-metrics-certs\") pod \"network-metrics-daemon-9flf9\" (UID: \"761dc4fd-5005-48a1-9874-bfa11455f308\") " pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:59:04.499216 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:04.499185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7697\""
Sep 17 17:59:04.507183 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:04.507154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9flf9"
Sep 17 17:59:04.564081 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:04.564051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:59:04.814314 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:04.814280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9flf9"]
Sep 17 17:59:04.827424 ip-10-0-143-209 kubenswrapper[2577]: W0917 17:59:04.827398    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761dc4fd_5005_48a1_9874_bfa11455f308.slice/crio-6cf72a199fc99cc6ae8a3e8a31f25813a23e3612c114ff4d4a4afac44ecd29a6 WatchSource:0}: Error finding container 6cf72a199fc99cc6ae8a3e8a31f25813a23e3612c114ff4d4a4afac44ecd29a6: Status 404 returned error can't find the container with id 6cf72a199fc99cc6ae8a3e8a31f25813a23e3612c114ff4d4a4afac44ecd29a6
Sep 17 17:59:05.281626 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:05.281593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9flf9" event={"ID":"761dc4fd-5005-48a1-9874-bfa11455f308","Type":"ContainerStarted","Data":"6cf72a199fc99cc6ae8a3e8a31f25813a23e3612c114ff4d4a4afac44ecd29a6"}
Sep 17 17:59:06.269300 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:06.269272    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bf5dbb977-nbqmw"
Sep 17 17:59:07.295681 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:07.295638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9flf9" event={"ID":"761dc4fd-5005-48a1-9874-bfa11455f308","Type":"ContainerStarted","Data":"766a8ab956a59a41713562b771236f5dc734559e648b0e1d6ff32843bae53bcd"}
Sep 17 17:59:07.296170 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:07.295697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9flf9" event={"ID":"761dc4fd-5005-48a1-9874-bfa11455f308","Type":"ContainerStarted","Data":"c701b43f526e3e7fee361e78c476eb2da34dcbed98bf23652f90d2c8bba6eb4e"}
Sep 17 17:59:07.320026 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:07.319974    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9flf9" podStartSLOduration=130.704732548 podStartE2EDuration="2m12.319956055s" totalImagesPullingTime="1.615223507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:07.318705663 +0000 UTC m=+134.161868810" watchObservedRunningTime="2026-09-17 17:59:07.319956055 +0000 UTC m=+134.163119203"
Sep 17 17:59:09.281464 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:09.281423    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:59:09.281904 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:09.281523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:59:09.286439 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:09.286418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:59:09.307266 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:09.307243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 17:59:12.668728 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:12.668666    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bbbc46cc7-rzbjp" podUID="028158c0-fe2c-4a67-9a58-e861cfceae7b" containerName="console" containerID="cri-o://a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d" gracePeriod=15
Sep 17 17:59:12.948479 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:12.948446    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:59:13.002887 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.002857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-serving-cert\") pod \"028158c0-fe2c-4a67-9a58-e861cfceae7b\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") "
Sep 17 17:59:13.003055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.002896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptpfd\" (UniqueName: \"kubernetes.io/projected/028158c0-fe2c-4a67-9a58-e861cfceae7b-kube-api-access-ptpfd\") pod \"028158c0-fe2c-4a67-9a58-e861cfceae7b\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") "
Sep 17 17:59:13.003055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.002922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-oauth-config\") pod \"028158c0-fe2c-4a67-9a58-e861cfceae7b\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") "
Sep 17 17:59:13.003055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.002948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-oauth-serving-cert\") pod \"028158c0-fe2c-4a67-9a58-e861cfceae7b\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") "
Sep 17 17:59:13.003055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.002968    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-config\") pod \"028158c0-fe2c-4a67-9a58-e861cfceae7b\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") "
Sep 17 17:59:13.003055 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.002993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-service-ca\") pod \"028158c0-fe2c-4a67-9a58-e861cfceae7b\" (UID: \"028158c0-fe2c-4a67-9a58-e861cfceae7b\") "
Sep 17 17:59:13.003410 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.003383    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-config" (OuterVolumeSpecName: "console-config") pod "028158c0-fe2c-4a67-9a58-e861cfceae7b" (UID: "028158c0-fe2c-4a67-9a58-e861cfceae7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:13.003410 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.003396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "028158c0-fe2c-4a67-9a58-e861cfceae7b" (UID: "028158c0-fe2c-4a67-9a58-e861cfceae7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:13.003498 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.003432    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "028158c0-fe2c-4a67-9a58-e861cfceae7b" (UID: "028158c0-fe2c-4a67-9a58-e861cfceae7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:13.005276 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.005255    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "028158c0-fe2c-4a67-9a58-e861cfceae7b" (UID: "028158c0-fe2c-4a67-9a58-e861cfceae7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:13.005436 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.005303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028158c0-fe2c-4a67-9a58-e861cfceae7b-kube-api-access-ptpfd" (OuterVolumeSpecName: "kube-api-access-ptpfd") pod "028158c0-fe2c-4a67-9a58-e861cfceae7b" (UID: "028158c0-fe2c-4a67-9a58-e861cfceae7b"). InnerVolumeSpecName "kube-api-access-ptpfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:13.005436 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.005345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "028158c0-fe2c-4a67-9a58-e861cfceae7b" (UID: "028158c0-fe2c-4a67-9a58-e861cfceae7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:13.104134 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.104107    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:13.104134 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.104129    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-service-ca\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:13.104134 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.104138    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:13.104312 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.104147    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptpfd\" (UniqueName: \"kubernetes.io/projected/028158c0-fe2c-4a67-9a58-e861cfceae7b-kube-api-access-ptpfd\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:13.104312 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.104155    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/028158c0-fe2c-4a67-9a58-e861cfceae7b-console-oauth-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:13.104312 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.104164    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/028158c0-fe2c-4a67-9a58-e861cfceae7b-oauth-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:13.323039 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.322952    2577 generic.go:356] "Generic (PLEG): container finished" podID="028158c0-fe2c-4a67-9a58-e861cfceae7b" containerID="a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d" exitCode=0
Sep 17 17:59:13.323039 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.322995    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbbc46cc7-rzbjp"
Sep 17 17:59:13.323039 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.323013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbbc46cc7-rzbjp" event={"ID":"028158c0-fe2c-4a67-9a58-e861cfceae7b","Type":"ContainerDied","Data":"a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d"}
Sep 17 17:59:13.323256 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.323082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbbc46cc7-rzbjp" event={"ID":"028158c0-fe2c-4a67-9a58-e861cfceae7b","Type":"ContainerDied","Data":"eea9b54dee666c208bef376000345efcf9d6981b0ee56ea13fcf4648869144a4"}
Sep 17 17:59:13.323256 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.323104    2577 scope.go:117] "RemoveContainer" containerID="a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d"
Sep 17 17:59:13.332125 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.332105    2577 scope.go:117] "RemoveContainer" containerID="a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d"
Sep 17 17:59:13.332400 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:59:13.332374    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d\": container with ID starting with a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d not found: ID does not exist" containerID="a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d"
Sep 17 17:59:13.332484 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.332405    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d"} err="failed to get container status \"a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d\": rpc error: code = NotFound desc = could not find container \"a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d\": container with ID starting with a808ed08c103c06b9070b58d9ce5c944f4a5650a40e5cb03efcccab88ebc459d not found: ID does not exist"
Sep 17 17:59:13.352432 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.352390    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbbc46cc7-rzbjp"]
Sep 17 17:59:13.355271 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.355253    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bbbc46cc7-rzbjp"]
Sep 17 17:59:13.584080 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:13.584009    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="028158c0-fe2c-4a67-9a58-e861cfceae7b" path="/var/lib/kubelet/pods/028158c0-fe2c-4a67-9a58-e861cfceae7b/volumes"
Sep 17 17:59:16.337780 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:16.337748    2577 generic.go:356] "Generic (PLEG): container finished" podID="8cbb56b5-5062-4840-8036-e7a2ca11f98a" containerID="e79430ff2c2cb295ee54be7d333d65ccc79e0eb59a8fd4b58d8e3a2a07117130" exitCode=0
Sep 17 17:59:16.338155 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:16.337817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92" event={"ID":"8cbb56b5-5062-4840-8036-e7a2ca11f98a","Type":"ContainerDied","Data":"e79430ff2c2cb295ee54be7d333d65ccc79e0eb59a8fd4b58d8e3a2a07117130"}
Sep 17 17:59:16.338155 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:16.338113    2577 scope.go:117] "RemoveContainer" containerID="e79430ff2c2cb295ee54be7d333d65ccc79e0eb59a8fd4b58d8e3a2a07117130"
Sep 17 17:59:17.343857 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:17.343824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnd92" event={"ID":"8cbb56b5-5062-4840-8036-e7a2ca11f98a","Type":"ContainerStarted","Data":"777cdd6db0652b28b67db7f17c7e82df0da0345ebc1a233b5202e09fa4fb7320"}
Sep 17 17:59:23.865223 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:23.865177    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d89c9995d-q2j5p" podUID="1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" containerName="console" containerID="cri-o://382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f" gracePeriod=15
Sep 17 17:59:24.153579 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.153538    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:59:24.299649 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-config\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.299818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-trusted-ca-bundle\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.299818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-service-ca\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.299818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgz4v\" (UniqueName: \"kubernetes.io/projected/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-kube-api-access-pgz4v\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.299818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-serving-cert\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.299818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-oauth-serving-cert\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.299818 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.299809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-oauth-config\") pod \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\" (UID: \"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1\") "
Sep 17 17:59:24.300150 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.300112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-config" (OuterVolumeSpecName: "console-config") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:24.300262 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.300229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:24.300262 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.300246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:24.300262 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.300258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:24.301952 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.301922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-kube-api-access-pgz4v" (OuterVolumeSpecName: "kube-api-access-pgz4v") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "kube-api-access-pgz4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:24.302049 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.301950    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:24.302102 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.302036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" (UID: "1974a01e-8c72-4dbf-a9c8-7944f2bf54e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:24.374983 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.374900    2577 generic.go:356] "Generic (PLEG): container finished" podID="1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" containerID="382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f" exitCode=0
Sep 17 17:59:24.374983 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.374950    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d89c9995d-q2j5p"
Sep 17 17:59:24.375148 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.374992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89c9995d-q2j5p" event={"ID":"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1","Type":"ContainerDied","Data":"382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f"}
Sep 17 17:59:24.375148 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.375044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d89c9995d-q2j5p" event={"ID":"1974a01e-8c72-4dbf-a9c8-7944f2bf54e1","Type":"ContainerDied","Data":"135f4a2022fecc6c5f64806000714e27a5c8cd5e6ff277ec9b10e85f25b810be"}
Sep 17 17:59:24.375148 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.375064    2577 scope.go:117] "RemoveContainer" containerID="382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f"
Sep 17 17:59:24.383537 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.383519    2577 scope.go:117] "RemoveContainer" containerID="382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f"
Sep 17 17:59:24.383802 ip-10-0-143-209 kubenswrapper[2577]: E0917 17:59:24.383781    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f\": container with ID starting with 382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f not found: ID does not exist" containerID="382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f"
Sep 17 17:59:24.383850 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.383811    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f"} err="failed to get container status \"382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f\": rpc error: code = NotFound desc = could not find container \"382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f\": container with ID starting with 382ad9ba78e182904c5afa3dae17ef08f50b68d5a4a4dcc0832b01411bdc162f not found: ID does not exist"
Sep 17 17:59:24.400615 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400590    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-oauth-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.400615 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400613    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-oauth-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.400801 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400623    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.400801 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400632    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-trusted-ca-bundle\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.400801 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400640    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-service-ca\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.400801 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400647    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgz4v\" (UniqueName: \"kubernetes.io/projected/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-kube-api-access-pgz4v\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.400801 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400656    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1-console-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 17:59:24.401021 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.400964    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d89c9995d-q2j5p"]
Sep 17 17:59:24.408320 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:24.408296    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d89c9995d-q2j5p"]
Sep 17 17:59:25.584469 ip-10-0-143-209 kubenswrapper[2577]: I0917 17:59:25.584431    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" path="/var/lib/kubelet/pods/1974a01e-8c72-4dbf-a9c8-7944f2bf54e1/volumes"
Sep 17 18:00:10.561525 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561486    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:10.562057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561817    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="alertmanager" containerID="cri-o://d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9" gracePeriod=120
Sep 17 18:00:10.562057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561851    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy" containerID="cri-o://42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6" gracePeriod=120
Sep 17 18:00:10.562057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561859    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="prom-label-proxy" containerID="cri-o://7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb" gracePeriod=120
Sep 17 18:00:10.562057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561902    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-web" containerID="cri-o://da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab" gracePeriod=120
Sep 17 18:00:10.562057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561912    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-metric" containerID="cri-o://300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77" gracePeriod=120
Sep 17 18:00:10.562057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:10.561937    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="config-reloader" containerID="cri-o://0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134" gracePeriod=120
Sep 17 18:00:11.579783 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.579753    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb" exitCode=0
Sep 17 18:00:11.579783 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.579778    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6" exitCode=0
Sep 17 18:00:11.579783 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.579785    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134" exitCode=0
Sep 17 18:00:11.579783 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.579790    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9" exitCode=0
Sep 17 18:00:11.583258 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.583232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"}
Sep 17 18:00:11.583358 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.583292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"}
Sep 17 18:00:11.583358 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.583319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"}
Sep 17 18:00:11.583358 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.583343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"}
Sep 17 18:00:11.812146 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.812123    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:11.891469 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-out\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891469 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-tls-assets\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891469 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891498    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t49f8\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-kube-api-access-t49f8\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-metrics-client-ca\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-main-db\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-trusted-ca-bundle\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.891748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-main-tls\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.892098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-volume\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.892098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891801    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-web-config\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.892098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.891821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-cluster-tls-config\") pod \"39c758a6-22ff-47d4-a78f-039e83ef13a6\" (UID: \"39c758a6-22ff-47d4-a78f-039e83ef13a6\") "
Sep 17 18:00:11.893087 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.892382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:11.893087 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.892681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:11.893751 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.893693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:11.894363 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.894337    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-out" (OuterVolumeSpecName: "config-out") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:11.894805 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.894779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.895684 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.895645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.896407 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.896366    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-volume" (OuterVolumeSpecName: "config-volume") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.896531 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.896502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:11.896621 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.896600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.896684 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.896628    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.897417 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.897388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-kube-api-access-t49f8" (OuterVolumeSpecName: "kube-api-access-t49f8") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "kube-api-access-t49f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:11.899972 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.899952    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.909580 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.909537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-web-config" (OuterVolumeSpecName: "web-config") pod "39c758a6-22ff-47d4-a78f-039e83ef13a6" (UID: "39c758a6-22ff-47d4-a78f-039e83ef13a6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.993214 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993176    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-web-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993214 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993210    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-cluster-tls-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993214 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993221    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-out\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993230    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-tls-assets\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993240    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993249    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993258    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t49f8\" (UniqueName: \"kubernetes.io/projected/39c758a6-22ff-47d4-a78f-039e83ef13a6-kube-api-access-t49f8\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993267    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-metrics-client-ca\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993275    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993283    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-main-db\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993291    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c758a6-22ff-47d4-a78f-039e83ef13a6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993299    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-secret-alertmanager-main-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.993461 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:11.993308    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39c758a6-22ff-47d4-a78f-039e83ef13a6-config-volume\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:12.586399 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586370    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77" exitCode=0
Sep 17 18:00:12.586399 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586394    2577 generic.go:356] "Generic (PLEG): container finished" podID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerID="da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab" exitCode=0
Sep 17 18:00:12.586858 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"}
Sep 17 18:00:12.586858 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586440    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.586858 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"}
Sep 17 18:00:12.586858 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"39c758a6-22ff-47d4-a78f-039e83ef13a6","Type":"ContainerDied","Data":"b4546e30ac9cebeee2df98e058543e8a2ba3aab802e086377a85b51b49fc4dcd"}
Sep 17 18:00:12.586858 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.586534    2577 scope.go:117] "RemoveContainer" containerID="7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"
Sep 17 18:00:12.594507 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.594492    2577 scope.go:117] "RemoveContainer" containerID="300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"
Sep 17 18:00:12.601241 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.601226    2577 scope.go:117] "RemoveContainer" containerID="42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"
Sep 17 18:00:12.607759 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.607742    2577 scope.go:117] "RemoveContainer" containerID="da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"
Sep 17 18:00:12.612290 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.612265    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:12.614909 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.614797    2577 scope.go:117] "RemoveContainer" containerID="0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"
Sep 17 18:00:12.618684 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.618662    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:12.622218 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.622202    2577 scope.go:117] "RemoveContainer" containerID="d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"
Sep 17 18:00:12.629119 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.629103    2577 scope.go:117] "RemoveContainer" containerID="b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"
Sep 17 18:00:12.635730 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.635709    2577 scope.go:117] "RemoveContainer" containerID="7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"
Sep 17 18:00:12.635989 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.635973    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb\": container with ID starting with 7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb not found: ID does not exist" containerID="7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"
Sep 17 18:00:12.636030 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.635999    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"} err="failed to get container status \"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb\": rpc error: code = NotFound desc = could not find container \"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb\": container with ID starting with 7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb not found: ID does not exist"
Sep 17 18:00:12.636030 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636016    2577 scope.go:117] "RemoveContainer" containerID="300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"
Sep 17 18:00:12.636235 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.636219    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77\": container with ID starting with 300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77 not found: ID does not exist" containerID="300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"
Sep 17 18:00:12.636277 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636240    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"} err="failed to get container status \"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77\": rpc error: code = NotFound desc = could not find container \"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77\": container with ID starting with 300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77 not found: ID does not exist"
Sep 17 18:00:12.636277 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636254    2577 scope.go:117] "RemoveContainer" containerID="42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"
Sep 17 18:00:12.636491 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.636472    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6\": container with ID starting with 42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6 not found: ID does not exist" containerID="42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"
Sep 17 18:00:12.636531 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636497    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"} err="failed to get container status \"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6\": rpc error: code = NotFound desc = could not find container \"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6\": container with ID starting with 42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6 not found: ID does not exist"
Sep 17 18:00:12.636531 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636514    2577 scope.go:117] "RemoveContainer" containerID="da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"
Sep 17 18:00:12.636751 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.636736    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab\": container with ID starting with da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab not found: ID does not exist" containerID="da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"
Sep 17 18:00:12.636789 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636753    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"} err="failed to get container status \"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab\": rpc error: code = NotFound desc = could not find container \"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab\": container with ID starting with da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab not found: ID does not exist"
Sep 17 18:00:12.636789 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636765    2577 scope.go:117] "RemoveContainer" containerID="0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"
Sep 17 18:00:12.636989 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.636974    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134\": container with ID starting with 0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134 not found: ID does not exist" containerID="0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"
Sep 17 18:00:12.637029 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.636992    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"} err="failed to get container status \"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134\": rpc error: code = NotFound desc = could not find container \"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134\": container with ID starting with 0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134 not found: ID does not exist"
Sep 17 18:00:12.637029 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637004    2577 scope.go:117] "RemoveContainer" containerID="d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"
Sep 17 18:00:12.637237 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.637222    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9\": container with ID starting with d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9 not found: ID does not exist" containerID="d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"
Sep 17 18:00:12.637280 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637239    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"} err="failed to get container status \"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9\": rpc error: code = NotFound desc = could not find container \"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9\": container with ID starting with d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9 not found: ID does not exist"
Sep 17 18:00:12.637280 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637252    2577 scope.go:117] "RemoveContainer" containerID="b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"
Sep 17 18:00:12.637491 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:12.637472    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad\": container with ID starting with b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad not found: ID does not exist" containerID="b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"
Sep 17 18:00:12.637637 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637498    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"} err="failed to get container status \"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad\": rpc error: code = NotFound desc = could not find container \"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad\": container with ID starting with b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad not found: ID does not exist"
Sep 17 18:00:12.637637 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637518    2577 scope.go:117] "RemoveContainer" containerID="7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"
Sep 17 18:00:12.637785 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637767    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb"} err="failed to get container status \"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb\": rpc error: code = NotFound desc = could not find container \"7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb\": container with ID starting with 7b1f284180013765b0dbeda18ef25f2fe226beda9f3b8734c7ea2e92391e87cb not found: ID does not exist"
Sep 17 18:00:12.637828 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637785    2577 scope.go:117] "RemoveContainer" containerID="300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"
Sep 17 18:00:12.638004 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.637988    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77"} err="failed to get container status \"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77\": rpc error: code = NotFound desc = could not find container \"300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77\": container with ID starting with 300341aeaf8536d8a7f58f056117eb7a624b64ad9422100d10e0375aa877ab77 not found: ID does not exist"
Sep 17 18:00:12.638056 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638005    2577 scope.go:117] "RemoveContainer" containerID="42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"
Sep 17 18:00:12.638231 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638213    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6"} err="failed to get container status \"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6\": rpc error: code = NotFound desc = could not find container \"42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6\": container with ID starting with 42592309f7b6b29b310f581d7885ff7689c0f7da3935f2d74e42f7ae538e25c6 not found: ID does not exist"
Sep 17 18:00:12.638297 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638232    2577 scope.go:117] "RemoveContainer" containerID="da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"
Sep 17 18:00:12.638485 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638463    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab"} err="failed to get container status \"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab\": rpc error: code = NotFound desc = could not find container \"da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab\": container with ID starting with da56b11932ec2887dec97e6f9d9d536c26ef53ea9fb1ab3dc5a0eea91219c5ab not found: ID does not exist"
Sep 17 18:00:12.638485 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638483    2577 scope.go:117] "RemoveContainer" containerID="0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"
Sep 17 18:00:12.638730 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638712    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134"} err="failed to get container status \"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134\": rpc error: code = NotFound desc = could not find container \"0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134\": container with ID starting with 0e556b79b91b159a08e8413fe6ae09b7a590496f9e574b39c4cad392e554b134 not found: ID does not exist"
Sep 17 18:00:12.638782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638730    2577 scope.go:117] "RemoveContainer" containerID="d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"
Sep 17 18:00:12.638925 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638911    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9"} err="failed to get container status \"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9\": rpc error: code = NotFound desc = could not find container \"d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9\": container with ID starting with d7de573485525ec14fadcd73aa82fbf9e7df54a06c06f366452ff857cb137cc9 not found: ID does not exist"
Sep 17 18:00:12.638964 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.638925    2577 scope.go:117] "RemoveContainer" containerID="b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"
Sep 17 18:00:12.639135 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.639121    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad"} err="failed to get container status \"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad\": rpc error: code = NotFound desc = could not find container \"b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad\": container with ID starting with b584b5dc1b2ba1f42b350a53fdfb535d256f001656e048529b47f006fc0d0dad not found: ID does not exist"
Sep 17 18:00:12.657922 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.657897    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:12.658178 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658168    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-web"
Sep 17 18:00:12.658220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658180    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-web"
Sep 17 18:00:12.658220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658189    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy"
Sep 17 18:00:12.658220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy"
Sep 17 18:00:12.658220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658201    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="prom-label-proxy"
Sep 17 18:00:12.658220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658206    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="prom-label-proxy"
Sep 17 18:00:12.658220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658219    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="028158c0-fe2c-4a67-9a58-e861cfceae7b" containerName="console"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658224    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="028158c0-fe2c-4a67-9a58-e861cfceae7b" containerName="console"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658236    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658243    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="init-config-reloader"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658247    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="init-config-reloader"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658253    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="alertmanager"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658258    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="alertmanager"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658264    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" containerName="console"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" containerName="console"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658277    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="config-reloader"
Sep 17 18:00:12.658386 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658281    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="config-reloader"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658757    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-web"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658778    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1974a01e-8c72-4dbf-a9c8-7944f2bf54e1" containerName="console"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658823    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="prom-label-proxy"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658834    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658853    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="alertmanager"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658862    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="config-reloader"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658870    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:12.658899 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.658889    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="028158c0-fe2c-4a67-9a58-e861cfceae7b" containerName="console"
Sep 17 18:00:12.666163 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.666141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.669377 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.669355    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:00:12.669377 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.669370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:00:12.670044 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.670023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:00:12.670420 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.670390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:00:12.670496 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.670473    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r9sgp\""
Sep 17 18:00:12.670580 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.670521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:00:12.672204 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.672186    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:00:12.672298 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.672238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:00:12.673273 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.673255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:00:12.676721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.676702    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:00:12.680925 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.680903    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:12.801478 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801478 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmwl\" (UniqueName: \"kubernetes.io/projected/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-kube-api-access-2vmwl\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801721 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801996 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801996 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-web-config\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.801996 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.801832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-config-out\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.902633 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmwl\" (UniqueName: \"kubernetes.io/projected/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-kube-api-access-2vmwl\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.902807 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.902807 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.902807 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.902807 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-web-config\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.902807 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-config-out\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.902943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903374 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.903350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.903496 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.903479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.904071 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.903422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.904071 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.903888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.904516 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.904496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.905977 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.905899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.906103 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.906078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.906313 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.906287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-config-out\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.906494 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.906477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.906605 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.906582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.906706 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.906683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.907235 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.907221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-web-config\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.907445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.907425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.907741 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.907727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.912210 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.912186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmwl\" (UniqueName: \"kubernetes.io/projected/5c1c2c4b-d5cf-455a-a27e-b75e297b303b-kube-api-access-2vmwl\") pod \"alertmanager-main-0\" (UID: \"5c1c2c4b-d5cf-455a-a27e-b75e297b303b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:12.977955 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:12.977920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:13.260099 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:13.260017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:13.264130 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:00:13.264104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1c2c4b_d5cf_455a_a27e_b75e297b303b.slice/crio-39d9b901b595ecb29ecafbe75fdb8c0c7e211ea0106addedd798678fb63b676d WatchSource:0}: Error finding container 39d9b901b595ecb29ecafbe75fdb8c0c7e211ea0106addedd798678fb63b676d: Status 404 returned error can't find the container with id 39d9b901b595ecb29ecafbe75fdb8c0c7e211ea0106addedd798678fb63b676d
Sep 17 18:00:13.584824 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:13.584749    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c758a6-22ff-47d4-a78f-039e83ef13a6" path="/var/lib/kubelet/pods/39c758a6-22ff-47d4-a78f-039e83ef13a6/volumes"
Sep 17 18:00:13.591852 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:13.591819    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c1c2c4b-d5cf-455a-a27e-b75e297b303b" containerID="3cc2b1a956eea2cabf7b31a9be6d45e441927c9192135629f4c1c28ad338891e" exitCode=0
Sep 17 18:00:13.592213 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:13.591893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerDied","Data":"3cc2b1a956eea2cabf7b31a9be6d45e441927c9192135629f4c1c28ad338891e"}
Sep 17 18:00:13.592213 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:13.591959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"39d9b901b595ecb29ecafbe75fdb8c0c7e211ea0106addedd798678fb63b676d"}
Sep 17 18:00:14.602610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.601078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"61a0915b6f0f461a0a4438166e2f972087935efd12ae309ab83f5e7fbad11ea8"}
Sep 17 18:00:14.602610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.601171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"535416fb6347b52f272bd89f31b8301d90b2d45c377a36251c76f0523f06f465"}
Sep 17 18:00:14.602610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.601216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"fce6c03a7c3fe03b4d8b2dd7ad733a5935dbdf08a646aff7d63fe1d580265d8d"}
Sep 17 18:00:14.602610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.601254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"a9562f8ccdda6ba7aca464cd0e0a5e4c6d27f9b016361a24b73a5d153e2c6449"}
Sep 17 18:00:14.602610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.601291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"e8142f44a041546386983b81c2b6aaea490570c22238d75460d6c95e5f0b24f8"}
Sep 17 18:00:14.602610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.601331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c1c2c4b-d5cf-455a-a27e-b75e297b303b","Type":"ContainerStarted","Data":"0925b6ac710e20f55b112662e3b14e85c07dce6ce27bd3fe9fc2436280f8e7e8"}
Sep 17 18:00:14.635718 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:14.634098    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.620130867 podStartE2EDuration="2.634074185s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.943318ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:14.633174202 +0000 UTC m=+201.476337361" watchObservedRunningTime="2026-09-17 18:00:14.634074185 +0000 UTC m=+201.477237334"
Sep 17 18:00:19.336017 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.335978    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67846c89-fz4lg"]
Sep 17 18:00:19.338544 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.338524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.352350 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.352324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67846c89-fz4lg"]
Sep 17 18:00:19.460343 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-service-ca\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.460515 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-oauth-config\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.460515 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-oauth-serving-cert\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.460515 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-config\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.460515 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jkk\" (UniqueName: \"kubernetes.io/projected/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-kube-api-access-z6jkk\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.460700 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-trusted-ca-bundle\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.460700 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.460596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-serving-cert\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.561717 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.561684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-trusted-ca-bundle\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.561929 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.561729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-serving-cert\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.561929 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.561757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-service-ca\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.561929 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.561776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-oauth-config\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562100 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.561956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-oauth-serving-cert\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562100 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.562023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-config\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562100 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.562067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jkk\" (UniqueName: \"kubernetes.io/projected/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-kube-api-access-z6jkk\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562510 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.562485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-service-ca\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562739 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.562717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-config\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562817 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.562738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-oauth-serving-cert\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.562817 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.562800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-trusted-ca-bundle\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.564277 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.564255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-serving-cert\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.564373 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.564289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-oauth-config\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.571855 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.571836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jkk\" (UniqueName: \"kubernetes.io/projected/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-kube-api-access-z6jkk\") pod \"console-67846c89-fz4lg\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") " pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.648630 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.648513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:19.921971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:19.921942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67846c89-fz4lg"]
Sep 17 18:00:19.924011 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:00:19.923982    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc049a1_24b7_4ae6_a3c1_be36c93dc383.slice/crio-9d3bf8dbb7717395d848ab7262f4e5f6f2840aaff92c15048055f264056c34eb WatchSource:0}: Error finding container 9d3bf8dbb7717395d848ab7262f4e5f6f2840aaff92c15048055f264056c34eb: Status 404 returned error can't find the container with id 9d3bf8dbb7717395d848ab7262f4e5f6f2840aaff92c15048055f264056c34eb
Sep 17 18:00:20.629531 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:20.629497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67846c89-fz4lg" event={"ID":"4dc049a1-24b7-4ae6-a3c1-be36c93dc383","Type":"ContainerStarted","Data":"a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884"}
Sep 17 18:00:20.629982 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:20.629544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67846c89-fz4lg" event={"ID":"4dc049a1-24b7-4ae6-a3c1-be36c93dc383","Type":"ContainerStarted","Data":"9d3bf8dbb7717395d848ab7262f4e5f6f2840aaff92c15048055f264056c34eb"}
Sep 17 18:00:20.649433 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:20.649380    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67846c89-fz4lg" podStartSLOduration=1.649364348 podStartE2EDuration="1.649364348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:20.64805743 +0000 UTC m=+207.491220579" watchObservedRunningTime="2026-09-17 18:00:20.649364348 +0000 UTC m=+207.492527496"
Sep 17 18:00:29.649414 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:29.649378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:29.649836 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:29.649450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:29.653811 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:29.653789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:29.669099 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:29.669076    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:00:29.727624 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:29.727593    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d568fc8d-dn4s6"]
Sep 17 18:00:54.751761 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:54.751717    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58d568fc8d-dn4s6" podUID="86166a37-8a0e-40eb-8cbf-c21386e76d6e" containerName="console" containerID="cri-o://f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5" gracePeriod=15
Sep 17 18:00:55.000206 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.000175    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 18:00:55.044082 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.043997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-oauth-serving-cert\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044082 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-oauth-config\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044297 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s44rf\" (UniqueName: \"kubernetes.io/projected/86166a37-8a0e-40eb-8cbf-c21386e76d6e-kube-api-access-s44rf\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044297 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-serving-cert\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044297 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-config\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044297 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-service-ca\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-trusted-ca-bundle\") pod \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\" (UID: \"86166a37-8a0e-40eb-8cbf-c21386e76d6e\") "
Sep 17 18:00:55.044502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:55.044634 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044555    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-oauth-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.044874 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-config" (OuterVolumeSpecName: "console-config") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:55.044874 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-service-ca" (OuterVolumeSpecName: "service-ca") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:55.045045 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.044955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:55.046437 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.046413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86166a37-8a0e-40eb-8cbf-c21386e76d6e-kube-api-access-s44rf" (OuterVolumeSpecName: "kube-api-access-s44rf") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "kube-api-access-s44rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:55.046715 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.046694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:55.046767 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.046715    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "86166a37-8a0e-40eb-8cbf-c21386e76d6e" (UID: "86166a37-8a0e-40eb-8cbf-c21386e76d6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:55.145724 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.145688    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-trusted-ca-bundle\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.145724 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.145718    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-oauth-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.145724 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.145730    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s44rf\" (UniqueName: \"kubernetes.io/projected/86166a37-8a0e-40eb-8cbf-c21386e76d6e-kube-api-access-s44rf\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.145971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.145740    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.145971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.145750    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-console-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.145971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.145758    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86166a37-8a0e-40eb-8cbf-c21386e76d6e-service-ca\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:00:55.775534 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.775448    2577 generic.go:356] "Generic (PLEG): container finished" podID="86166a37-8a0e-40eb-8cbf-c21386e76d6e" containerID="f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5" exitCode=0
Sep 17 18:00:55.775534 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.775491    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d568fc8d-dn4s6"
Sep 17 18:00:55.776035 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.775536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d568fc8d-dn4s6" event={"ID":"86166a37-8a0e-40eb-8cbf-c21386e76d6e","Type":"ContainerDied","Data":"f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5"}
Sep 17 18:00:55.776035 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.775604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d568fc8d-dn4s6" event={"ID":"86166a37-8a0e-40eb-8cbf-c21386e76d6e","Type":"ContainerDied","Data":"7c9d3c888a2347942e1b7c7b391bb86561020bb0b4c90f179a06ac40bf08b023"}
Sep 17 18:00:55.776035 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.775626    2577 scope.go:117] "RemoveContainer" containerID="f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5"
Sep 17 18:00:55.783692 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.783674    2577 scope.go:117] "RemoveContainer" containerID="f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5"
Sep 17 18:00:55.783986 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:00:55.783966    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5\": container with ID starting with f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5 not found: ID does not exist" containerID="f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5"
Sep 17 18:00:55.784062 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.783991    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5"} err="failed to get container status \"f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5\": rpc error: code = NotFound desc = could not find container \"f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5\": container with ID starting with f1aee103d0f5ac396693e9fa51e196b74b81468550f111c5efa1216ed692ebe5 not found: ID does not exist"
Sep 17 18:00:55.794722 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.794687    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d568fc8d-dn4s6"]
Sep 17 18:00:55.801473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:55.801447    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58d568fc8d-dn4s6"]
Sep 17 18:00:57.585801 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:00:57.585763    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86166a37-8a0e-40eb-8cbf-c21386e76d6e" path="/var/lib/kubelet/pods/86166a37-8a0e-40eb-8cbf-c21386e76d6e/volumes"
Sep 17 18:01:23.834310 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.834233    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"]
Sep 17 18:01:23.834766 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.834589    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86166a37-8a0e-40eb-8cbf-c21386e76d6e" containerName="console"
Sep 17 18:01:23.834766 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.834603    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86166a37-8a0e-40eb-8cbf-c21386e76d6e" containerName="console"
Sep 17 18:01:23.834766 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.834695    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86166a37-8a0e-40eb-8cbf-c21386e76d6e" containerName="console"
Sep 17 18:01:23.837012 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.836994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:23.841294 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.841267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:01:23.841599 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.841340    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l7j57\""
Sep 17 18:01:23.842497 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.842473    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:01:23.860872 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.860847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"]
Sep 17 18:01:23.975424 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.975396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:23.975582 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.975439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98gh\" (UniqueName: \"kubernetes.io/projected/21437560-8b3e-405b-b68a-880f3dcbf20e-kube-api-access-m98gh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:23.975582 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:23.975467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.076266 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.076238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.076398 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.076279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98gh\" (UniqueName: \"kubernetes.io/projected/21437560-8b3e-405b-b68a-880f3dcbf20e-kube-api-access-m98gh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.076398 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.076307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.076686 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.076666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.076686 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.076677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.086610 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.086546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98gh\" (UniqueName: \"kubernetes.io/projected/21437560-8b3e-405b-b68a-880f3dcbf20e-kube-api-access-m98gh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.148516 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.148495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:24.421058 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.421034    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"]
Sep 17 18:01:24.423533 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:01:24.423504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21437560_8b3e_405b_b68a_880f3dcbf20e.slice/crio-e900981dc25763f09a95b4eef200390824fc6796f8a318b249a7721060c96f3d WatchSource:0}: Error finding container e900981dc25763f09a95b4eef200390824fc6796f8a318b249a7721060c96f3d: Status 404 returned error can't find the container with id e900981dc25763f09a95b4eef200390824fc6796f8a318b249a7721060c96f3d
Sep 17 18:01:24.898442 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:24.898358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" event={"ID":"21437560-8b3e-405b-b68a-880f3dcbf20e","Type":"ContainerStarted","Data":"e900981dc25763f09a95b4eef200390824fc6796f8a318b249a7721060c96f3d"}
Sep 17 18:01:31.931295 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:31.931264    2577 generic.go:356] "Generic (PLEG): container finished" podID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerID="ecb7da797f8267fe0174ea63c8726818378f4e32358e9b949aeab767bea1be1f" exitCode=0
Sep 17 18:01:31.931701 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:31.931319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" event={"ID":"21437560-8b3e-405b-b68a-880f3dcbf20e","Type":"ContainerDied","Data":"ecb7da797f8267fe0174ea63c8726818378f4e32358e9b949aeab767bea1be1f"}
Sep 17 18:01:37.956874 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:37.956842    2577 generic.go:356] "Generic (PLEG): container finished" podID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerID="852db410d08be66114040993a60e8d1b3b02d8b364d4cc1018aba5dd6e33bdf5" exitCode=0
Sep 17 18:01:37.957341 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:37.956926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" event={"ID":"21437560-8b3e-405b-b68a-880f3dcbf20e","Type":"ContainerDied","Data":"852db410d08be66114040993a60e8d1b3b02d8b364d4cc1018aba5dd6e33bdf5"}
Sep 17 18:01:45.995265 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:45.995231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" event={"ID":"21437560-8b3e-405b-b68a-880f3dcbf20e","Type":"ContainerStarted","Data":"a83f57557ab7f2c9ba43e4577f9ab108353d8188b45cbe3a38e1f5e26bc32f74"}
Sep 17 18:01:46.025344 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:46.025296    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" podStartSLOduration=2.610190021 podStartE2EDuration="23.025280051s" totalImagesPullingTime="20.41509003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:46.022977007 +0000 UTC m=+292.866140158" watchObservedRunningTime="2026-09-17 18:01:46.025280051 +0000 UTC m=+292.868443198"
Sep 17 18:01:47.001142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:47.001112    2577 generic.go:356] "Generic (PLEG): container finished" podID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerID="a83f57557ab7f2c9ba43e4577f9ab108353d8188b45cbe3a38e1f5e26bc32f74" exitCode=0
Sep 17 18:01:47.001514 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:47.001199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" event={"ID":"21437560-8b3e-405b-b68a-880f3dcbf20e","Type":"ContainerDied","Data":"a83f57557ab7f2c9ba43e4577f9ab108353d8188b45cbe3a38e1f5e26bc32f74"}
Sep 17 18:01:48.130754 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.130732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:48.190009 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.189980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m98gh\" (UniqueName: \"kubernetes.io/projected/21437560-8b3e-405b-b68a-880f3dcbf20e-kube-api-access-m98gh\") pod \"21437560-8b3e-405b-b68a-880f3dcbf20e\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") "
Sep 17 18:01:48.190144 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.190022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-util\") pod \"21437560-8b3e-405b-b68a-880f3dcbf20e\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") "
Sep 17 18:01:48.190144 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.190053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-bundle\") pod \"21437560-8b3e-405b-b68a-880f3dcbf20e\" (UID: \"21437560-8b3e-405b-b68a-880f3dcbf20e\") "
Sep 17 18:01:48.190838 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.190811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-bundle" (OuterVolumeSpecName: "bundle") pod "21437560-8b3e-405b-b68a-880f3dcbf20e" (UID: "21437560-8b3e-405b-b68a-880f3dcbf20e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:48.192150 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.192130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21437560-8b3e-405b-b68a-880f3dcbf20e-kube-api-access-m98gh" (OuterVolumeSpecName: "kube-api-access-m98gh") pod "21437560-8b3e-405b-b68a-880f3dcbf20e" (UID: "21437560-8b3e-405b-b68a-880f3dcbf20e"). InnerVolumeSpecName "kube-api-access-m98gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:48.194873 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.194848    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-util" (OuterVolumeSpecName: "util") pod "21437560-8b3e-405b-b68a-880f3dcbf20e" (UID: "21437560-8b3e-405b-b68a-880f3dcbf20e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:48.290958 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.290893    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-util\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:01:48.290958 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.290916    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21437560-8b3e-405b-b68a-880f3dcbf20e-bundle\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:01:48.290958 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:48.290926    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98gh\" (UniqueName: \"kubernetes.io/projected/21437560-8b3e-405b-b68a-880f3dcbf20e-kube-api-access-m98gh\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.012017 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:49.011982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv" event={"ID":"21437560-8b3e-405b-b68a-880f3dcbf20e","Type":"ContainerDied","Data":"e900981dc25763f09a95b4eef200390824fc6796f8a318b249a7721060c96f3d"}
Sep 17 18:01:49.012177 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:49.012027    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e900981dc25763f09a95b4eef200390824fc6796f8a318b249a7721060c96f3d"
Sep 17 18:01:49.012177 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:49.011989    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1at8rzv"
Sep 17 18:01:52.458169 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458131    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"]
Sep 17 18:01:52.458727 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="pull"
Sep 17 18:01:52.458802 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458729    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="pull"
Sep 17 18:01:52.458802 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458745    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="extract"
Sep 17 18:01:52.458802 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458752    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="extract"
Sep 17 18:01:52.458802 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458774    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="util"
Sep 17 18:01:52.458802 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458783    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="util"
Sep 17 18:01:52.459035 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.458871    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="21437560-8b3e-405b-b68a-880f3dcbf20e" containerName="extract"
Sep 17 18:01:52.460983 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.460963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.466126 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.465909    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:01:52.466126 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.465949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:01:52.466126 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.466024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-976dk\""
Sep 17 18:01:52.466348 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.466284    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:01:52.483292 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.483271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"]
Sep 17 18:01:52.523850 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.523820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b705e2f0-79f1-4311-a4e2-896be3bf29aa-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2smt5\" (UID: \"b705e2f0-79f1-4311-a4e2-896be3bf29aa\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.523965 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.523856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xhnl\" (UniqueName: \"kubernetes.io/projected/b705e2f0-79f1-4311-a4e2-896be3bf29aa-kube-api-access-8xhnl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2smt5\" (UID: \"b705e2f0-79f1-4311-a4e2-896be3bf29aa\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.624976 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.624944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b705e2f0-79f1-4311-a4e2-896be3bf29aa-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2smt5\" (UID: \"b705e2f0-79f1-4311-a4e2-896be3bf29aa\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.624976 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.624976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xhnl\" (UniqueName: \"kubernetes.io/projected/b705e2f0-79f1-4311-a4e2-896be3bf29aa-kube-api-access-8xhnl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2smt5\" (UID: \"b705e2f0-79f1-4311-a4e2-896be3bf29aa\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.627404 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.627379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b705e2f0-79f1-4311-a4e2-896be3bf29aa-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2smt5\" (UID: \"b705e2f0-79f1-4311-a4e2-896be3bf29aa\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.642263 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.642234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xhnl\" (UniqueName: \"kubernetes.io/projected/b705e2f0-79f1-4311-a4e2-896be3bf29aa-kube-api-access-8xhnl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2smt5\" (UID: \"b705e2f0-79f1-4311-a4e2-896be3bf29aa\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:52.770546 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:52.770463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:53.080780 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:53.080705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"]
Sep 17 18:01:53.083957 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:01:53.083931    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb705e2f0_79f1_4311_a4e2_896be3bf29aa.slice/crio-d4301de295a2e152f5e2cf8324b8fb1fa7c1a183dc0276ae667834bb726cfcd9 WatchSource:0}: Error finding container d4301de295a2e152f5e2cf8324b8fb1fa7c1a183dc0276ae667834bb726cfcd9: Status 404 returned error can't find the container with id d4301de295a2e152f5e2cf8324b8fb1fa7c1a183dc0276ae667834bb726cfcd9
Sep 17 18:01:54.035064 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:54.035023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5" event={"ID":"b705e2f0-79f1-4311-a4e2-896be3bf29aa","Type":"ContainerStarted","Data":"d4301de295a2e152f5e2cf8324b8fb1fa7c1a183dc0276ae667834bb726cfcd9"}
Sep 17 18:01:55.723799 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:55.723777    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:01:55.724711 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:55.724693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:01:55.733909 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:55.733881    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:01:56.045319 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:56.045224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5" event={"ID":"b705e2f0-79f1-4311-a4e2-896be3bf29aa","Type":"ContainerStarted","Data":"bc0b84940c7ac32dad1de1994d61b468aaa460e3ebb1ddcb9cc5f5b29e2a628a"}
Sep 17 18:01:56.045486 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:56.045330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:01:56.072146 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:56.072105    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5" podStartSLOduration=1.419160677 podStartE2EDuration="4.072093234s" totalImagesPullingTime="2.652932557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:56.070220013 +0000 UTC m=+302.913383162" watchObservedRunningTime="2026-09-17 18:01:56.072093234 +0000 UTC m=+302.915256381"
Sep 17 18:01:57.724912 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.724871    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nm424"]
Sep 17 18:01:57.727300 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.727281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:57.731075 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.731050    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:01:57.731183 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.731080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:01:57.731183 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.731050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-m6tsg\""
Sep 17 18:01:57.758898 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.758866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nm424"]
Sep 17 18:01:57.766264 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.766231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-certificates\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:57.766403 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.766283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6nxz\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-kube-api-access-p6nxz\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:57.867896 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.867092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-certificates\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:57.867896 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.867165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6nxz\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-kube-api-access-p6nxz\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:57.867896 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:01:57.867640    2577 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 18:01:57.867896 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:01:57.867666    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-nm424: secret "keda-admission-webhooks-certs" not found
Sep 17 18:01:57.867896 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:01:57.867724    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-certificates podName:f3bf5389-07f2-4446-b8ed-c0b99ff58bc8 nodeName:}" failed. No retries permitted until 2026-09-17 18:01:58.367704763 +0000 UTC m=+305.210867891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-certificates") pod "keda-admission-6c4d864b86-nm424" (UID: "f3bf5389-07f2-4446-b8ed-c0b99ff58bc8") : secret "keda-admission-webhooks-certs" not found
Sep 17 18:01:57.884342 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:57.884311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6nxz\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-kube-api-access-p6nxz\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:58.371428 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:58.371390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-certificates\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:58.373867 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:58.373848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3bf5389-07f2-4446-b8ed-c0b99ff58bc8-certificates\") pod \"keda-admission-6c4d864b86-nm424\" (UID: \"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8\") " pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:58.637346 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:58.637269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:01:58.924553 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:58.924527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nm424"]
Sep 17 18:01:58.926998 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:01:58.926974    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bf5389_07f2_4446_b8ed_c0b99ff58bc8.slice/crio-0c66515a889e8f422d5cd0791ccc137f0c7ae15fa20dd3cde26dce2d37d67a42 WatchSource:0}: Error finding container 0c66515a889e8f422d5cd0791ccc137f0c7ae15fa20dd3cde26dce2d37d67a42: Status 404 returned error can't find the container with id 0c66515a889e8f422d5cd0791ccc137f0c7ae15fa20dd3cde26dce2d37d67a42
Sep 17 18:01:58.928283 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:58.928260    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:01:59.060753 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:01:59.060720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nm424" event={"ID":"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8","Type":"ContainerStarted","Data":"0c66515a889e8f422d5cd0791ccc137f0c7ae15fa20dd3cde26dce2d37d67a42"}
Sep 17 18:02:01.071274 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:02:01.071241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nm424" event={"ID":"f3bf5389-07f2-4446-b8ed-c0b99ff58bc8","Type":"ContainerStarted","Data":"3143aeb6a889d43cc7bfeb86e13c8377e33c7e1352759091e0c81aea70bb4c55"}
Sep 17 18:02:01.071666 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:02:01.071308    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:02:01.118538 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:02:01.118487    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-nm424" podStartSLOduration=2.898519329 podStartE2EDuration="4.118472477s" totalImagesPullingTime="1.219953148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:01.117101861 +0000 UTC m=+307.960265009" watchObservedRunningTime="2026-09-17 18:02:01.118472477 +0000 UTC m=+307.961635625"
Sep 17 18:02:17.052638 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:02:17.052610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2smt5"
Sep 17 18:02:22.078238 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:02:22.078211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-nm424"
Sep 17 18:03:19.252920 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.252888    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"]
Sep 17 18:03:19.255266 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.255244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.258116 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.258090    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:03:19.258116 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.258100    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:03:19.259219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.259199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7d6hn\""
Sep 17 18:03:19.259321 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.259243    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:03:19.275136 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.272265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"]
Sep 17 18:03:19.287335 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.287306    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qm5xs"]
Sep 17 18:03:19.289533 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.289516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.292125 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.292102    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:03:19.292232 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.292184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-t6pqr\""
Sep 17 18:03:19.299622 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.299599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qm5xs"]
Sep 17 18:03:19.305083 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.305057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2gt\" (UniqueName: \"kubernetes.io/projected/0cfa7477-453f-4ba0-a378-98cbec28f6e6-kube-api-access-xf2gt\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.305183 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.305104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfa7477-453f-4ba0-a378-98cbec28f6e6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.305245 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.305212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnn5\" (UniqueName: \"kubernetes.io/projected/48dc32db-3c72-4aaa-afc3-93176e9f414b-kube-api-access-fhnn5\") pod \"seaweedfs-64568bcd49-qm5xs\" (UID: \"48dc32db-3c72-4aaa-afc3-93176e9f414b\") " pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.305304 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.305251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48dc32db-3c72-4aaa-afc3-93176e9f414b-data\") pod \"seaweedfs-64568bcd49-qm5xs\" (UID: \"48dc32db-3c72-4aaa-afc3-93176e9f414b\") " pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.405599 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.405547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfa7477-453f-4ba0-a378-98cbec28f6e6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.405747 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.405649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnn5\" (UniqueName: \"kubernetes.io/projected/48dc32db-3c72-4aaa-afc3-93176e9f414b-kube-api-access-fhnn5\") pod \"seaweedfs-64568bcd49-qm5xs\" (UID: \"48dc32db-3c72-4aaa-afc3-93176e9f414b\") " pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.405747 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.405683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48dc32db-3c72-4aaa-afc3-93176e9f414b-data\") pod \"seaweedfs-64568bcd49-qm5xs\" (UID: \"48dc32db-3c72-4aaa-afc3-93176e9f414b\") " pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.405747 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:03:19.405698    2577 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 18:03:19.405747 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.405715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2gt\" (UniqueName: \"kubernetes.io/projected/0cfa7477-453f-4ba0-a378-98cbec28f6e6-kube-api-access-xf2gt\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.405954 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:03:19.405774    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfa7477-453f-4ba0-a378-98cbec28f6e6-cert podName:0cfa7477-453f-4ba0-a378-98cbec28f6e6 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:19.905754362 +0000 UTC m=+386.748917508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfa7477-453f-4ba0-a378-98cbec28f6e6-cert") pod "llmisvc-controller-manager-645f9c8fd6-trgsr" (UID: "0cfa7477-453f-4ba0-a378-98cbec28f6e6") : secret "llmisvc-webhook-server-cert" not found
Sep 17 18:03:19.406096 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.406073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48dc32db-3c72-4aaa-afc3-93176e9f414b-data\") pod \"seaweedfs-64568bcd49-qm5xs\" (UID: \"48dc32db-3c72-4aaa-afc3-93176e9f414b\") " pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.414816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.414795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2gt\" (UniqueName: \"kubernetes.io/projected/0cfa7477-453f-4ba0-a378-98cbec28f6e6-kube-api-access-xf2gt\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.414914 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.414899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnn5\" (UniqueName: \"kubernetes.io/projected/48dc32db-3c72-4aaa-afc3-93176e9f414b-kube-api-access-fhnn5\") pod \"seaweedfs-64568bcd49-qm5xs\" (UID: \"48dc32db-3c72-4aaa-afc3-93176e9f414b\") " pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.600376 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.600277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:19.872938 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.872868    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qm5xs"]
Sep 17 18:03:19.876165 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:03:19.876135    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dc32db_3c72_4aaa_afc3_93176e9f414b.slice/crio-29d409ea2c07496e412bc4f1616cd3ee049c8d2c03d96adab35f40c62803dc56 WatchSource:0}: Error finding container 29d409ea2c07496e412bc4f1616cd3ee049c8d2c03d96adab35f40c62803dc56: Status 404 returned error can't find the container with id 29d409ea2c07496e412bc4f1616cd3ee049c8d2c03d96adab35f40c62803dc56
Sep 17 18:03:19.911154 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.911128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfa7477-453f-4ba0-a378-98cbec28f6e6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:19.913459 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:19.913439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfa7477-453f-4ba0-a378-98cbec28f6e6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-trgsr\" (UID: \"0cfa7477-453f-4ba0-a378-98cbec28f6e6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:20.165720 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:20.165681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:20.423855 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:20.423776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qm5xs" event={"ID":"48dc32db-3c72-4aaa-afc3-93176e9f414b","Type":"ContainerStarted","Data":"29d409ea2c07496e412bc4f1616cd3ee049c8d2c03d96adab35f40c62803dc56"}
Sep 17 18:03:20.442802 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:20.442732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"]
Sep 17 18:03:20.445487 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:03:20.445459    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0cfa7477_453f_4ba0_a378_98cbec28f6e6.slice/crio-710e110b694315802e25f21cbcdb69eb7c79624c86ab6c57cf42a4740df19719 WatchSource:0}: Error finding container 710e110b694315802e25f21cbcdb69eb7c79624c86ab6c57cf42a4740df19719: Status 404 returned error can't find the container with id 710e110b694315802e25f21cbcdb69eb7c79624c86ab6c57cf42a4740df19719
Sep 17 18:03:21.431699 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:21.431654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr" event={"ID":"0cfa7477-453f-4ba0-a378-98cbec28f6e6","Type":"ContainerStarted","Data":"710e110b694315802e25f21cbcdb69eb7c79624c86ab6c57cf42a4740df19719"}
Sep 17 18:03:24.448554 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:24.448518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qm5xs" event={"ID":"48dc32db-3c72-4aaa-afc3-93176e9f414b","Type":"ContainerStarted","Data":"c6e5be060099ca1f8dfe175ff9d175924467e1cc13265053da356144e316070e"}
Sep 17 18:03:24.449314 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:24.448632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:24.450322 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:24.450298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr" event={"ID":"0cfa7477-453f-4ba0-a378-98cbec28f6e6","Type":"ContainerStarted","Data":"160c67c979e2f098120427712e60d3abdf60581373bc4d6cf4967705d9b97911"}
Sep 17 18:03:24.450415 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:24.450367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:03:24.470101 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:24.470060    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qm5xs" podStartSLOduration=1.461076855 podStartE2EDuration="5.470047486s" totalImagesPullingTime="4.008970631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:24.46717236 +0000 UTC m=+391.310335509" watchObservedRunningTime="2026-09-17 18:03:24.470047486 +0000 UTC m=+391.313210633"
Sep 17 18:03:24.486832 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:24.486795    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr" podStartSLOduration=2.118401999 podStartE2EDuration="5.486781796s" totalImagesPullingTime="3.368379797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:24.486253534 +0000 UTC m=+391.329416684" watchObservedRunningTime="2026-09-17 18:03:24.486781796 +0000 UTC m=+391.329944983"
Sep 17 18:03:30.457347 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:30.457320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qm5xs"
Sep 17 18:03:33.504251 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.504220    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f66f5db7-nkbgs"]
Sep 17 18:03:33.507246 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.507230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.528484 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.528453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f66f5db7-nkbgs"]
Sep 17 18:03:33.623073 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-service-ca\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.623239 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-config\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.623239 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vk7d\" (UniqueName: \"kubernetes.io/projected/3a77fae6-da8b-4c65-aa16-f9ece05b053d-kube-api-access-8vk7d\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.623239 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-serving-cert\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.623341 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-oauth-config\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.623341 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-oauth-serving-cert\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.623341 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.623313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-trusted-ca-bundle\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724294 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-service-ca\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724425 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-config\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724425 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vk7d\" (UniqueName: \"kubernetes.io/projected/3a77fae6-da8b-4c65-aa16-f9ece05b053d-kube-api-access-8vk7d\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724425 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-serving-cert\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724425 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-oauth-config\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724655 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-oauth-serving-cert\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.724655 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.724609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-trusted-ca-bundle\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.725151 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.725124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-service-ca\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.725288 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.725203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-oauth-serving-cert\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.725359 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.725311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-trusted-ca-bundle\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.725483 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.725464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-config\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.726810 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.726790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-oauth-config\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.726943 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.726926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a77fae6-da8b-4c65-aa16-f9ece05b053d-console-serving-cert\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.735348 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.735330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vk7d\" (UniqueName: \"kubernetes.io/projected/3a77fae6-da8b-4c65-aa16-f9ece05b053d-kube-api-access-8vk7d\") pod \"console-86f66f5db7-nkbgs\" (UID: \"3a77fae6-da8b-4c65-aa16-f9ece05b053d\") " pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:33.816274 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:33.816211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:34.095892 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:34.095824    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f66f5db7-nkbgs"]
Sep 17 18:03:34.099084 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:03:34.099050    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a77fae6_da8b_4c65_aa16_f9ece05b053d.slice/crio-ba23ed0ca5fb9874d75c3d45866b39374f1bf1d8b41c0e088e2a720ac6cd8e43 WatchSource:0}: Error finding container ba23ed0ca5fb9874d75c3d45866b39374f1bf1d8b41c0e088e2a720ac6cd8e43: Status 404 returned error can't find the container with id ba23ed0ca5fb9874d75c3d45866b39374f1bf1d8b41c0e088e2a720ac6cd8e43
Sep 17 18:03:34.497034 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:34.496991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f66f5db7-nkbgs" event={"ID":"3a77fae6-da8b-4c65-aa16-f9ece05b053d","Type":"ContainerStarted","Data":"372ed4fee3e5ddab4c12ada346d42583c7ddf573453f517436a1d2912bf6817f"}
Sep 17 18:03:34.497237 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:34.497045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f66f5db7-nkbgs" event={"ID":"3a77fae6-da8b-4c65-aa16-f9ece05b053d","Type":"ContainerStarted","Data":"ba23ed0ca5fb9874d75c3d45866b39374f1bf1d8b41c0e088e2a720ac6cd8e43"}
Sep 17 18:03:34.520485 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:34.520436    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86f66f5db7-nkbgs" podStartSLOduration=1.520421843 podStartE2EDuration="1.520421843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:34.518792378 +0000 UTC m=+401.361955526" watchObservedRunningTime="2026-09-17 18:03:34.520421843 +0000 UTC m=+401.363584992"
Sep 17 18:03:43.816810 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:43.816768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:43.817193 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:43.816933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:43.821882 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:43.821864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:44.551003 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:44.550974    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f66f5db7-nkbgs"
Sep 17 18:03:44.631412 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:44.631379    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67846c89-fz4lg"]
Sep 17 18:03:55.457694 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:03:55.457666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-trgsr"
Sep 17 18:04:09.650591 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:09.650484    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67846c89-fz4lg" podUID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" containerName="console" containerID="cri-o://a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884" gracePeriod=15
Sep 17 18:04:09.666104 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:09.666074    2577 patch_prober.go:28] interesting pod/console-67846c89-fz4lg container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.25:8443/health\": dial tcp 10.133.0.25:8443: connect: connection refused" start-of-body=
Sep 17 18:04:09.666212 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:09.666138    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-67846c89-fz4lg" podUID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" containerName="console" probeResult="failure" output="Get \"https://10.133.0.25:8443/health\": dial tcp 10.133.0.25:8443: connect: connection refused"
Sep 17 18:04:09.885578 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:09.885541    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:04:10.014522 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014448    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-serving-cert\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.014522 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-service-ca\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.014522 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-oauth-serving-cert\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.014816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-oauth-config\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.014816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014600    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-trusted-ca-bundle\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.014816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-config\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.014816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6jkk\" (UniqueName: \"kubernetes.io/projected/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-kube-api-access-z6jkk\") pod \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\" (UID: \"4dc049a1-24b7-4ae6-a3c1-be36c93dc383\") "
Sep 17 18:04:10.015021 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:10.015021 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.014937    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-service-ca" (OuterVolumeSpecName: "service-ca") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:10.015098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.015018    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-service-ca\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.015098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.015036    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-oauth-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.015098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.015081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-config" (OuterVolumeSpecName: "console-config") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:10.015374 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.015345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:10.016768 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.016744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:10.016860 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.016799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-kube-api-access-z6jkk" (OuterVolumeSpecName: "kube-api-access-z6jkk") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "kube-api-access-z6jkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:10.016860 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.016819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4dc049a1-24b7-4ae6-a3c1-be36c93dc383" (UID: "4dc049a1-24b7-4ae6-a3c1-be36c93dc383"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:10.115445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.115408    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.115445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.115439    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jkk\" (UniqueName: \"kubernetes.io/projected/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-kube-api-access-z6jkk\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.115445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.115452    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-serving-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.115694 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.115464    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-console-oauth-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.115694 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.115475    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc049a1-24b7-4ae6-a3c1-be36c93dc383-trusted-ca-bundle\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:04:10.661584 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.661532    2577 generic.go:356] "Generic (PLEG): container finished" podID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" containerID="a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884" exitCode=0
Sep 17 18:04:10.662008 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.661591    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67846c89-fz4lg"
Sep 17 18:04:10.662008 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.661623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67846c89-fz4lg" event={"ID":"4dc049a1-24b7-4ae6-a3c1-be36c93dc383","Type":"ContainerDied","Data":"a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884"}
Sep 17 18:04:10.662008 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.661676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67846c89-fz4lg" event={"ID":"4dc049a1-24b7-4ae6-a3c1-be36c93dc383","Type":"ContainerDied","Data":"9d3bf8dbb7717395d848ab7262f4e5f6f2840aaff92c15048055f264056c34eb"}
Sep 17 18:04:10.662008 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.661703    2577 scope.go:117] "RemoveContainer" containerID="a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884"
Sep 17 18:04:10.670135 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.670120    2577 scope.go:117] "RemoveContainer" containerID="a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884"
Sep 17 18:04:10.670400 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:04:10.670384    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884\": container with ID starting with a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884 not found: ID does not exist" containerID="a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884"
Sep 17 18:04:10.670452 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.670409    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884"} err="failed to get container status \"a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884\": rpc error: code = NotFound desc = could not find container \"a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884\": container with ID starting with a6758539c08cce15eae1c171a80d47fae665b191c68365f76c6d0307d6151884 not found: ID does not exist"
Sep 17 18:04:10.685956 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.685932    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67846c89-fz4lg"]
Sep 17 18:04:10.694190 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:10.691362    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67846c89-fz4lg"]
Sep 17 18:04:11.585326 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:04:11.585295    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" path="/var/lib/kubelet/pods/4dc049a1-24b7-4ae6-a3c1-be36c93dc383/volumes"
Sep 17 18:06:25.657313 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.657281    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"]
Sep 17 18:06:25.657798 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.657653    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" containerName="console"
Sep 17 18:06:25.657798 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.657664    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" containerName="console"
Sep 17 18:06:25.657798 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.657719    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc049a1-24b7-4ae6-a3c1-be36c93dc383" containerName="console"
Sep 17 18:06:25.659779 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.659761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.664140 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.664120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:06:25.664363 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.664344    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:06:25.664457 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.664347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:06:25.664672 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.664655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:06:25.664912 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.664894    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:06:25.665637 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.665623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:06:25.666110 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.666098    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:06:25.675740 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.675720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"]
Sep 17 18:06:25.746876 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.746845    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"]
Sep 17 18:06:25.749501 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.749486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.752867 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.752849    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-cb079-predictor-serving-cert\""
Sep 17 18:06:25.752968 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.752880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-cb079-kube-rbac-proxy-sar-config\""
Sep 17 18:06:25.763717 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.763696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"]
Sep 17 18:06:25.787218 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.787188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5218ca2-26c0-4458-958c-bd90d502995e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.787330 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.787236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl44s\" (UniqueName: \"kubernetes.io/projected/a5218ca2-26c0-4458-958c-bd90d502995e-kube-api-access-sl44s\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.787330 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.787290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.787330 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.787326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5218ca2-26c0-4458-958c-bd90d502995e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.787478 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.787347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.887793 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnk7g\" (UniqueName: \"kubernetes.io/projected/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-api-access-bnk7g\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5218ca2-26c0-4458-958c-bd90d502995e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl44s\" (UniqueName: \"kubernetes.io/projected/a5218ca2-26c0-4458-958c-bd90d502995e-kube-api-access-sl44s\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.887985 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.887987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.888354 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.888008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.888354 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.888031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5218ca2-26c0-4458-958c-bd90d502995e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.888354 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.888344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5218ca2-26c0-4458-958c-bd90d502995e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.888641 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.888621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.888718 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.888642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.890474 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.890458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5218ca2-26c0-4458-958c-bd90d502995e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.897087 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.897064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl44s\" (UniqueName: \"kubernetes.io/projected/a5218ca2-26c0-4458-958c-bd90d502995e-kube-api-access-sl44s\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-qr8p6\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.972627 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.972540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:25.988939 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.988916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.989046 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.988965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnk7g\" (UniqueName: \"kubernetes.io/projected/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-api-access-bnk7g\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.989046 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.989006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.989154 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.989051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.989154 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.989113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.989781 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.989761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.989905 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.989881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.990055 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.990038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.991278 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.991262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:25.999818 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:25.999792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnk7g\" (UniqueName: \"kubernetes.io/projected/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-api-access-bnk7g\") pod \"isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:26.059098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:26.059066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:26.256658 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:26.256629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"]
Sep 17 18:06:26.259457 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:06:26.259431    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5218ca2_26c0_4458_958c_bd90d502995e.slice/crio-c8303d2f2a16932ffed912fe9c28128fe73025bcde8071aade1a097d3144e269 WatchSource:0}: Error finding container c8303d2f2a16932ffed912fe9c28128fe73025bcde8071aade1a097d3144e269: Status 404 returned error can't find the container with id c8303d2f2a16932ffed912fe9c28128fe73025bcde8071aade1a097d3144e269
Sep 17 18:06:26.274245 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:26.274213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerStarted","Data":"c8303d2f2a16932ffed912fe9c28128fe73025bcde8071aade1a097d3144e269"}
Sep 17 18:06:26.344317 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:06:26.344285    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a8f91a_95a3_4aea_b66d_9a71cfd0b5ee.slice/crio-a2e5af6c3568ff588486d8d2e6d93dae708d6787156df00f28f9856b202816d5 WatchSource:0}: Error finding container a2e5af6c3568ff588486d8d2e6d93dae708d6787156df00f28f9856b202816d5: Status 404 returned error can't find the container with id a2e5af6c3568ff588486d8d2e6d93dae708d6787156df00f28f9856b202816d5
Sep 17 18:06:26.344535 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:26.344518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"]
Sep 17 18:06:27.281219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:27.281182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerStarted","Data":"a2e5af6c3568ff588486d8d2e6d93dae708d6787156df00f28f9856b202816d5"}
Sep 17 18:06:30.298957 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:30.298919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerStarted","Data":"340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504"}
Sep 17 18:06:30.300307 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:30.300286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerStarted","Data":"097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd"}
Sep 17 18:06:33.313331 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:33.313305    2577 generic.go:356] "Generic (PLEG): container finished" podID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerID="340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504" exitCode=0
Sep 17 18:06:33.313682 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:33.313384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerDied","Data":"340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504"}
Sep 17 18:06:34.324613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:34.323716    2577 generic.go:356] "Generic (PLEG): container finished" podID="a5218ca2-26c0-4458-958c-bd90d502995e" containerID="097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd" exitCode=0
Sep 17 18:06:34.324613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:34.323802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerDied","Data":"097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd"}
Sep 17 18:06:49.439980 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:49.439874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerStarted","Data":"5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45"}
Sep 17 18:06:49.443521 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:49.443427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerStarted","Data":"7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244"}
Sep 17 18:06:52.462687 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:52.462650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerStarted","Data":"5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a"}
Sep 17 18:06:52.463130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:52.462736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:52.463130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:52.462815    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:52.464434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:52.464406    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:06:52.465071 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:52.465054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerStarted","Data":"6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe"}
Sep 17 18:06:52.490552 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:52.490499    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podStartSLOduration=3.201235148 podStartE2EDuration="27.490482797s" totalImagesPullingTime="20.340728572s" totalInitContainerRuntime="3.948519077s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:52.487486513 +0000 UTC m=+599.330649661" watchObservedRunningTime="2026-09-17 18:06:52.490482797 +0000 UTC m=+599.333645946"
Sep 17 18:06:53.470020 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:53.469986    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:06:54.477028 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:54.476996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerStarted","Data":"d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac"}
Sep 17 18:06:54.477507 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:54.477211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:54.477507 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:54.477350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:54.478488 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:54.478464    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:06:54.503255 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:54.503213    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podStartSLOduration=2.08551626 podStartE2EDuration="29.503200804s" totalImagesPullingTime="23.708468907s" totalInitContainerRuntime="3.709215637s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:25 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:54.500982672 +0000 UTC m=+601.344145820" watchObservedRunningTime="2026-09-17 18:06:54.503200804 +0000 UTC m=+601.346364017"
Sep 17 18:06:55.481679 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:55.481647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:55.482128 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:55.481753    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:06:55.482782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:55.482756    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:06:55.761431 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:55.761356    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:06:55.761431 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:55.761357    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:06:56.486769 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:56.486731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:06:56.487209 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:56.487146    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:06:56.491471 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:56.491451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:06:57.491115 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:57.491075    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:06:57.491550 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:57.491332    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:06:58.475455 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:58.475429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:06:58.475931 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:06:58.475904    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:07.491368 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:07.491273    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:07.491882 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:07.491768    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:07:08.476045 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:08.476009    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:17.491924 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:17.491876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:17.492412 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:17.492286    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:07:18.475822 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:18.475781    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:27.491511 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:27.491460    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:27.492006 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:27.491965    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:07:28.476101 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:28.476062    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:37.491707 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:37.491657    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:37.492165 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:37.492141    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:07:38.476826 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:38.476782    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:47.491670 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:47.491612    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:47.492169 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:47.492094    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:07:48.476302 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:48.476264    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:57.492017 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:57.491968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:57.492489 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:57.492327    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:07:58.476755 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:07:58.476712    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:07.491747 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:07.491712    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:07.492282 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:07.491936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:08.477210 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:08.477179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:08:20.238489 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:20.238452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"]
Sep 17 18:08:20.239216 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:20.238782    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" containerID="cri-o://5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45" gracePeriod=30
Sep 17 18:08:20.239216 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:20.238804    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" containerID="cri-o://6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe" gracePeriod=30
Sep 17 18:08:20.239216 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:20.238950    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" containerID="cri-o://d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac" gracePeriod=30
Sep 17 18:08:20.881265 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:20.881230    2577 generic.go:356] "Generic (PLEG): container finished" podID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerID="6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe" exitCode=2
Sep 17 18:08:20.881434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:20.881300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerDied","Data":"6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe"}
Sep 17 18:08:21.487432 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:21.487391    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:08:24.901496 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:24.901468    2577 generic.go:356] "Generic (PLEG): container finished" podID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerID="5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45" exitCode=0
Sep 17 18:08:24.901874 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:24.901519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerDied","Data":"5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45"}
Sep 17 18:08:26.487299 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:26.487255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:08:27.491847 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:27.491805    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:27.492229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:27.492095    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:31.487773 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:31.487729    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:08:31.488157 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:31.487853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:36.487366 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:36.487287    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:08:37.491465 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:37.491426    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:37.491869 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:37.491782    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:41.487208 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:41.487169    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:08:46.487241 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:46.487202    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:08:47.491976 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:47.491938    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:47.492390 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:47.492069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:47.492390 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:47.492298    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:47.492511 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:47.492431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:50.384673 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.384647    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:50.502803 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.502715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-proxy-tls\") pod \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") "
Sep 17 18:08:50.502803 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.502782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-cabundle-cert\") pod \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") "
Sep 17 18:08:50.502803 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.502801    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnk7g\" (UniqueName: \"kubernetes.io/projected/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-api-access-bnk7g\") pod \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") "
Sep 17 18:08:50.503066 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.502840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kserve-provision-location\") pod \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") "
Sep 17 18:08:50.503066 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.502870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-rbac-proxy-sar-config\") pod \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\" (UID: \"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee\") "
Sep 17 18:08:50.503250 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.503156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" (UID: "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:50.503299 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.503242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" (UID: "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:50.503335 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.503315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" (UID: "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:50.504953 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.504928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" (UID: "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:50.505065 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.505048    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-api-access-bnk7g" (OuterVolumeSpecName: "kube-api-access-bnk7g") pod "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" (UID: "29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee"). InnerVolumeSpecName "kube-api-access-bnk7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:50.604242 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.604199    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:50.604242 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.604237    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:50.604242 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.604247    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:50.604242 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.604256    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:50.604484 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:50.604264    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnk7g\" (UniqueName: \"kubernetes.io/projected/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee-kube-api-access-bnk7g\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:08:51.032244 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.032212    2577 generic.go:356] "Generic (PLEG): container finished" podID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerID="d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac" exitCode=0
Sep 17 18:08:51.032430 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.032275    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"
Sep 17 18:08:51.032430 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.032277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerDied","Data":"d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac"}
Sep 17 18:08:51.032430 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.032405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4" event={"ID":"29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee","Type":"ContainerDied","Data":"a2e5af6c3568ff588486d8d2e6d93dae708d6787156df00f28f9856b202816d5"}
Sep 17 18:08:51.032555 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.032436    2577 scope.go:117] "RemoveContainer" containerID="d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac"
Sep 17 18:08:51.041620 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.041598    2577 scope.go:117] "RemoveContainer" containerID="6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe"
Sep 17 18:08:51.048892 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.048876    2577 scope.go:117] "RemoveContainer" containerID="5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45"
Sep 17 18:08:51.056224 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.056208    2577 scope.go:117] "RemoveContainer" containerID="340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504"
Sep 17 18:08:51.063464 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.063446    2577 scope.go:117] "RemoveContainer" containerID="d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac"
Sep 17 18:08:51.063752 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:08:51.063731    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac\": container with ID starting with d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac not found: ID does not exist" containerID="d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac"
Sep 17 18:08:51.063821 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.063759    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac"} err="failed to get container status \"d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac\": rpc error: code = NotFound desc = could not find container \"d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac\": container with ID starting with d62061cc1aab462b910487d995c1f4c577c30f12f748aa782e5815a05129f7ac not found: ID does not exist"
Sep 17 18:08:51.063821 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.063775    2577 scope.go:117] "RemoveContainer" containerID="6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe"
Sep 17 18:08:51.064017 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:08:51.063999    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe\": container with ID starting with 6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe not found: ID does not exist" containerID="6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe"
Sep 17 18:08:51.064060 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.064024    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe"} err="failed to get container status \"6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe\": rpc error: code = NotFound desc = could not find container \"6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe\": container with ID starting with 6a570073d9b9da68929cc87b69ab2bdaf1a4be110c5a7e15609da36dc6186afe not found: ID does not exist"
Sep 17 18:08:51.064060 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.064052    2577 scope.go:117] "RemoveContainer" containerID="5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45"
Sep 17 18:08:51.064283 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:08:51.064264    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45\": container with ID starting with 5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45 not found: ID does not exist" containerID="5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45"
Sep 17 18:08:51.064335 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.064287    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45"} err="failed to get container status \"5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45\": rpc error: code = NotFound desc = could not find container \"5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45\": container with ID starting with 5a130ee98fddd69bef55eb981b5db56d94bb11cb089aff34cb969e87c1ffcf45 not found: ID does not exist"
Sep 17 18:08:51.064335 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.064298    2577 scope.go:117] "RemoveContainer" containerID="340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504"
Sep 17 18:08:51.064530 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:08:51.064503    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504\": container with ID starting with 340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504 not found: ID does not exist" containerID="340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504"
Sep 17 18:08:51.064640 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.064538    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504"} err="failed to get container status \"340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504\": rpc error: code = NotFound desc = could not find container \"340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504\": container with ID starting with 340dac0c9e6e6c59b80a956785fdca00c400b4aac3d34801f1b34bec9e5b6504 not found: ID does not exist"
Sep 17 18:08:51.081698 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.081679    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"]
Sep 17 18:08:51.088683 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.088665    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb079-predictor-8f6596664-zrvx4"]
Sep 17 18:08:51.585867 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:08:51.585837    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" path="/var/lib/kubelet/pods/29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee/volumes"
Sep 17 18:09:45.334797 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:45.334764    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"]
Sep 17 18:09:45.335329 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:45.334986    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" containerID="cri-o://7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244" gracePeriod=30
Sep 17 18:09:45.335329 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:45.335033    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kube-rbac-proxy" containerID="cri-o://5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a" gracePeriod=30
Sep 17 18:09:46.289150 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:46.289119    2577 generic.go:356] "Generic (PLEG): container finished" podID="a5218ca2-26c0-4458-958c-bd90d502995e" containerID="5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a" exitCode=2
Sep 17 18:09:46.289312 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:46.289192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerDied","Data":"5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a"}
Sep 17 18:09:48.470988 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:48.470947    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:09:48.476328 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:48.476303    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:09:49.984207 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:49.984182    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:09:50.084881 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.084799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5218ca2-26c0-4458-958c-bd90d502995e-proxy-tls\") pod \"a5218ca2-26c0-4458-958c-bd90d502995e\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") "
Sep 17 18:09:50.084881 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.084837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-cabundle-cert\") pod \"a5218ca2-26c0-4458-958c-bd90d502995e\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") "
Sep 17 18:09:50.085053 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.084928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5218ca2-26c0-4458-958c-bd90d502995e-kserve-provision-location\") pod \"a5218ca2-26c0-4458-958c-bd90d502995e\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") "
Sep 17 18:09:50.085053 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.084985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl44s\" (UniqueName: \"kubernetes.io/projected/a5218ca2-26c0-4458-958c-bd90d502995e-kube-api-access-sl44s\") pod \"a5218ca2-26c0-4458-958c-bd90d502995e\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") "
Sep 17 18:09:50.085053 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.085012    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-kube-rbac-proxy-sar-config\") pod \"a5218ca2-26c0-4458-958c-bd90d502995e\" (UID: \"a5218ca2-26c0-4458-958c-bd90d502995e\") "
Sep 17 18:09:50.085231 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.085209    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5218ca2-26c0-4458-958c-bd90d502995e" (UID: "a5218ca2-26c0-4458-958c-bd90d502995e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:50.085301 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.085215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5218ca2-26c0-4458-958c-bd90d502995e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5218ca2-26c0-4458-958c-bd90d502995e" (UID: "a5218ca2-26c0-4458-958c-bd90d502995e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:50.085436 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.085413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5218ca2-26c0-4458-958c-bd90d502995e" (UID: "a5218ca2-26c0-4458-958c-bd90d502995e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:50.086989 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.086962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5218ca2-26c0-4458-958c-bd90d502995e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5218ca2-26c0-4458-958c-bd90d502995e" (UID: "a5218ca2-26c0-4458-958c-bd90d502995e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:50.087085 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.087009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5218ca2-26c0-4458-958c-bd90d502995e-kube-api-access-sl44s" (OuterVolumeSpecName: "kube-api-access-sl44s") pod "a5218ca2-26c0-4458-958c-bd90d502995e" (UID: "a5218ca2-26c0-4458-958c-bd90d502995e"). InnerVolumeSpecName "kube-api-access-sl44s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:50.185872 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.185846    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5218ca2-26c0-4458-958c-bd90d502995e-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.185872 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.185870    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.185872 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.185880    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5218ca2-26c0-4458-958c-bd90d502995e-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.186083 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.185889    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl44s\" (UniqueName: \"kubernetes.io/projected/a5218ca2-26c0-4458-958c-bd90d502995e-kube-api-access-sl44s\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.186083 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.185898    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5218ca2-26c0-4458-958c-bd90d502995e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.309613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.309555    2577 generic.go:356] "Generic (PLEG): container finished" podID="a5218ca2-26c0-4458-958c-bd90d502995e" containerID="7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244" exitCode=0
Sep 17 18:09:50.309613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.309594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerDied","Data":"7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244"}
Sep 17 18:09:50.309859 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.309637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"
Sep 17 18:09:50.309859 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.309676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6" event={"ID":"a5218ca2-26c0-4458-958c-bd90d502995e","Type":"ContainerDied","Data":"c8303d2f2a16932ffed912fe9c28128fe73025bcde8071aade1a097d3144e269"}
Sep 17 18:09:50.309859 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.309712    2577 scope.go:117] "RemoveContainer" containerID="5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a"
Sep 17 18:09:50.318921 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.318905    2577 scope.go:117] "RemoveContainer" containerID="7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244"
Sep 17 18:09:50.326240 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.326224    2577 scope.go:117] "RemoveContainer" containerID="097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd"
Sep 17 18:09:50.333049 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.333033    2577 scope.go:117] "RemoveContainer" containerID="5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a"
Sep 17 18:09:50.333343 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.333319    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a\": container with ID starting with 5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a not found: ID does not exist" containerID="5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a"
Sep 17 18:09:50.333476 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.333351    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a"} err="failed to get container status \"5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a\": rpc error: code = NotFound desc = could not find container \"5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a\": container with ID starting with 5951b656375ddd29fc3f3d830615b6650b6a4b7122bf393a80990254842a170a not found: ID does not exist"
Sep 17 18:09:50.333476 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.333368    2577 scope.go:117] "RemoveContainer" containerID="7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244"
Sep 17 18:09:50.333705 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.333681    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244\": container with ID starting with 7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244 not found: ID does not exist" containerID="7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244"
Sep 17 18:09:50.333776 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.333713    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244"} err="failed to get container status \"7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244\": rpc error: code = NotFound desc = could not find container \"7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244\": container with ID starting with 7a329fd9680b96af34345e894af65eeff1f10c8b5689f50b01e916f424d0b244 not found: ID does not exist"
Sep 17 18:09:50.333776 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.333734    2577 scope.go:117] "RemoveContainer" containerID="097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd"
Sep 17 18:09:50.334136 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.334113    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd\": container with ID starting with 097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd not found: ID does not exist" containerID="097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd"
Sep 17 18:09:50.334236 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.334145    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd"} err="failed to get container status \"097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd\": rpc error: code = NotFound desc = could not find container \"097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd\": container with ID starting with 097b683e5d7133c0560435af281ef5d82bd9dac047e70d03282e03e0a3fa5dfd not found: ID does not exist"
Sep 17 18:09:50.339040 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.338993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"]
Sep 17 18:09:50.343973 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.343956    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-qr8p6"]
Sep 17 18:09:50.509806 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.509779    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"]
Sep 17 18:09:50.510142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510112    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container"
Sep 17 18:09:50.510142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510122    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container"
Sep 17 18:09:50.510142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510131    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent"
Sep 17 18:09:50.510142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510137    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent"
Sep 17 18:09:50.510142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510145    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510177    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="storage-initializer"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510183    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="storage-initializer"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510189    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kube-rbac-proxy"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kube-rbac-proxy"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510213    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="storage-initializer"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510218    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="storage-initializer"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510264    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kserve-container"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510273    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="agent"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510281    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kube-rbac-proxy"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510288    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" containerName="kserve-container"
Sep 17 18:09:50.510311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.510295    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a8f91a-95a3-4aea-b66d-9a71cfd0b5ee" containerName="kube-rbac-proxy"
Sep 17 18:09:50.513540 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.513522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.528764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528504    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:09:50.528764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528545    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:09:50.528764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528574    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:09:50.528764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528574    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:09:50.528764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528639    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:09:50.528764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:09:50.529144 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.528954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:09:50.535728 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.535704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"]
Sep 17 18:09:50.588309 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.588284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d63981d-842e-4d90-9564-c44236b6566d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.588309 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.588314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.588543 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.588371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.588543 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.588442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8zx\" (UniqueName: \"kubernetes.io/projected/9d63981d-842e-4d90-9564-c44236b6566d-kube-api-access-7c8zx\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.588543 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.588499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.590694 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.590641    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"]
Sep 17 18:09:50.593174 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.593161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.596469 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.596455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:09:50.604335 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.604315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"]
Sep 17 18:09:50.689917 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.689880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690093 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.689924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8zx\" (UniqueName: \"kubernetes.io/projected/9d63981d-842e-4d90-9564-c44236b6566d-kube-api-access-7c8zx\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690093 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.689952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.690093 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.689991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690093 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xpj\" (UniqueName: \"kubernetes.io/projected/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-api-access-f8xpj\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.690093 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.690330 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.690330 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d63981d-842e-4d90-9564-c44236b6566d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690330 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690463 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.690447    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:09:50.690500 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d63981d-842e-4d90-9564-c44236b6566d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690552 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.690506    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls podName:9d63981d-842e-4d90-9564-c44236b6566d nodeName:}" failed. No retries permitted until 2026-09-17 18:09:51.190486627 +0000 UTC m=+778.033649767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls") pod "isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" (UID: "9d63981d-842e-4d90-9564-c44236b6566d") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:09:50.690669 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.690817 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.690801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.701523 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.701496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8zx\" (UniqueName: \"kubernetes.io/projected/9d63981d-842e-4d90-9564-c44236b6566d-kube-api-access-7c8zx\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:50.791548 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.791513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.791725 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.791614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.791725 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.791656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xpj\" (UniqueName: \"kubernetes.io/projected/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-api-access-f8xpj\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.791725 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.791676    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:09:50.791836 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:09:50.791751    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls podName:89c63f14-ac92-4725-a4fe-3cbaaea071d2 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:51.291733718 +0000 UTC m=+778.134896858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls") pod "isvc-canary-stop-predictor-b84cdfc7-b47j6" (UID: "89c63f14-ac92-4725-a4fe-3cbaaea071d2") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:09:50.791836 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.791784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.791836 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.791820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.792120 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.792102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.792404 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.792383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.792446 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.792432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:50.803972 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:50.803943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xpj\" (UniqueName: \"kubernetes.io/projected/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-api-access-f8xpj\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:51.196238 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.196203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:51.198667 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.198645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-wfs4f\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:51.296862 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.296830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:51.299090 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.299068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-b47j6\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:51.422900 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.422854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:51.502763 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.502722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:51.584984 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.584954    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5218ca2-26c0-4458-958c-bd90d502995e" path="/var/lib/kubelet/pods/a5218ca2-26c0-4458-958c-bd90d502995e/volumes"
Sep 17 18:09:51.694363 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.694341    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"]
Sep 17 18:09:51.696340 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:09:51.696312    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d63981d_842e_4d90_9564_c44236b6566d.slice/crio-48df575966b9d403a2ab8f50de0cfa273a16b6812faea46118acb620a8872506 WatchSource:0}: Error finding container 48df575966b9d403a2ab8f50de0cfa273a16b6812faea46118acb620a8872506: Status 404 returned error can't find the container with id 48df575966b9d403a2ab8f50de0cfa273a16b6812faea46118acb620a8872506
Sep 17 18:09:51.779380 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:51.779354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"]
Sep 17 18:09:51.779862 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:09:51.779838    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c63f14_ac92_4725_a4fe_3cbaaea071d2.slice/crio-d2e707954642924fb808c0ad84ebe6167a69c5d710813b6f4c3cdbec90326665 WatchSource:0}: Error finding container d2e707954642924fb808c0ad84ebe6167a69c5d710813b6f4c3cdbec90326665: Status 404 returned error can't find the container with id d2e707954642924fb808c0ad84ebe6167a69c5d710813b6f4c3cdbec90326665
Sep 17 18:09:52.331463 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:52.324414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerStarted","Data":"f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0"}
Sep 17 18:09:52.331463 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:52.324590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerStarted","Data":"d2e707954642924fb808c0ad84ebe6167a69c5d710813b6f4c3cdbec90326665"}
Sep 17 18:09:52.333631 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:52.333600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerStarted","Data":"cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb"}
Sep 17 18:09:52.333744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:52.333662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerStarted","Data":"48df575966b9d403a2ab8f50de0cfa273a16b6812faea46118acb620a8872506"}
Sep 17 18:09:56.355868 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:56.355833    2577 generic.go:356] "Generic (PLEG): container finished" podID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerID="f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0" exitCode=0
Sep 17 18:09:56.356248 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:56.355909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerDied","Data":"f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0"}
Sep 17 18:09:56.357676 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:56.357598    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d63981d-842e-4d90-9564-c44236b6566d" containerID="cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb" exitCode=0
Sep 17 18:09:56.357760 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:56.357670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerDied","Data":"cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb"}
Sep 17 18:09:57.365938 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.365900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerStarted","Data":"af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd"}
Sep 17 18:09:57.366334 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.366012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerStarted","Data":"a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a"}
Sep 17 18:09:57.366334 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.366228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:57.368005 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.367986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerStarted","Data":"56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d"}
Sep 17 18:09:57.368108 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.368026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerStarted","Data":"8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af"}
Sep 17 18:09:57.368332 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.368319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:57.368368 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.368357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:09:57.369698 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.369675    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:57.390866 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.390829    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podStartSLOduration=3.779500134 podStartE2EDuration="7.390818627s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.611318493s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:57.389399044 +0000 UTC m=+784.232562192" watchObservedRunningTime="2026-09-17 18:09:57.390818627 +0000 UTC m=+784.233981775"
Sep 17 18:09:57.418108 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:57.418070    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podStartSLOduration=3.576534653 podStartE2EDuration="7.418059218s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841524565s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:57.416771451 +0000 UTC m=+784.259934600" watchObservedRunningTime="2026-09-17 18:09:57.418059218 +0000 UTC m=+784.261222420"
Sep 17 18:09:58.373574 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:58.373513    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:58.374041 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:58.373732    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:09:58.374947 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:58.374921    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:09:59.378446 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:09:59.378407    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:03.377624 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:03.377595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:10:03.378074 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:03.378044    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:04.384015 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:04.383988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:10:04.384499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:04.384474    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:10.779456 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.779379    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"]
Sep 17 18:10:10.783070 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.783036    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"]
Sep 17 18:10:10.783229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.783169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:10.786225 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.786203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7da3c-predictor-serving-cert\""
Sep 17 18:10:10.786225 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.786215    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7da3c-kube-rbac-proxy-sar-config\""
Sep 17 18:10:10.858140 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.858105    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"]
Sep 17 18:10:10.860825 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.860810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:10.876714 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.876693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7da3c-predictor-serving-cert\""
Sep 17 18:10:10.876861 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.876841    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7da3c-kube-rbac-proxy-sar-config\""
Sep 17 18:10:10.878798 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.878778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"]
Sep 17 18:10:10.965432 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:10.965432 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb1a5a2-6151-4bdc-9568-002c61edb068-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb1a5a2-6151-4bdc-9568-002c61edb068-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wc8b\" (UniqueName: \"kubernetes.io/projected/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-api-access-5wc8b\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2fc54b-3b28-4f25-a085-766376c72102-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:10.965709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:10.965918 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:10.965711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czfn\" (UniqueName: \"kubernetes.io/projected/5f2fc54b-3b28-4f25-a085-766376c72102-kube-api-access-5czfn\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.066301 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.066301 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb1a5a2-6151-4bdc-9568-002c61edb068-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb1a5a2-6151-4bdc-9568-002c61edb068-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wc8b\" (UniqueName: \"kubernetes.io/projected/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-api-access-5wc8b\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2fc54b-3b28-4f25-a085-766376c72102-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.066499 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.066893 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czfn\" (UniqueName: \"kubernetes.io/projected/5f2fc54b-3b28-4f25-a085-766376c72102-kube-api-access-5czfn\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.066893 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.066769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb1a5a2-6151-4bdc-9568-002c61edb068-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.066893 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:10:11.066876    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-7da3c-predictor-serving-cert" not found
Sep 17 18:10:11.067041 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:10:11.066933    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls podName:5f2fc54b-3b28-4f25-a085-766376c72102 nodeName:}" failed. No retries permitted until 2026-09-17 18:10:11.566914568 +0000 UTC m=+798.410077699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" (UID: "5f2fc54b-3b28-4f25-a085-766376c72102") : secret "isvc-xgboost-graph-raw-hpa-7da3c-predictor-serving-cert" not found
Sep 17 18:10:11.067106 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.067050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.067166 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.067149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.067205 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.067148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2fc54b-3b28-4f25-a085-766376c72102-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.067291 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.067268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.067387 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.067370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.069106 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.069087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb1a5a2-6151-4bdc-9568-002c61edb068-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.076033 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.076008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wc8b\" (UniqueName: \"kubernetes.io/projected/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-api-access-5wc8b\") pod \"isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.076127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.076106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czfn\" (UniqueName: \"kubernetes.io/projected/5f2fc54b-3b28-4f25-a085-766376c72102-kube-api-access-5czfn\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.096841 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.096820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:11.371501 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.371476    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"]
Sep 17 18:10:11.373741 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:10:11.373715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb1a5a2_6151_4bdc_9568_002c61edb068.slice/crio-c284c3dc1fc48a04dc2027c2c32433f9c284c8f6706c731ea0ff9213a5aece26 WatchSource:0}: Error finding container c284c3dc1fc48a04dc2027c2c32433f9c284c8f6706c731ea0ff9213a5aece26: Status 404 returned error can't find the container with id c284c3dc1fc48a04dc2027c2c32433f9c284c8f6706c731ea0ff9213a5aece26
Sep 17 18:10:11.438798 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.438764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerStarted","Data":"1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a"}
Sep 17 18:10:11.438896 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.438842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerStarted","Data":"c284c3dc1fc48a04dc2027c2c32433f9c284c8f6706c731ea0ff9213a5aece26"}
Sep 17 18:10:11.570526 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.570486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.572887 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.572863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:11.770512 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:11.770470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:12.056760 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:12.056725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"]
Sep 17 18:10:12.057526 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:10:12.057498    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2fc54b_3b28_4f25_a085_766376c72102.slice/crio-b4c32a850e58042ed31f848751a2ef3d8ae3f84827d28174593668e7ea3dac31 WatchSource:0}: Error finding container b4c32a850e58042ed31f848751a2ef3d8ae3f84827d28174593668e7ea3dac31: Status 404 returned error can't find the container with id b4c32a850e58042ed31f848751a2ef3d8ae3f84827d28174593668e7ea3dac31
Sep 17 18:10:12.446915 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:12.446881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerStarted","Data":"af8d9251b0fc477bb5551abb6da303d53ddb9908f2f6ce53be1920b2a09fc067"}
Sep 17 18:10:12.447121 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:12.446951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerStarted","Data":"b4c32a850e58042ed31f848751a2ef3d8ae3f84827d28174593668e7ea3dac31"}
Sep 17 18:10:13.378332 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:13.378297    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:14.385337 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:14.385300    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:15.464312 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:15.464283    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerID="1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a" exitCode=0
Sep 17 18:10:15.464800 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:15.464353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerDied","Data":"1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a"}
Sep 17 18:10:16.470131 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.470103    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f2fc54b-3b28-4f25-a085-766376c72102" containerID="af8d9251b0fc477bb5551abb6da303d53ddb9908f2f6ce53be1920b2a09fc067" exitCode=0
Sep 17 18:10:16.470508 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.470177    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerDied","Data":"af8d9251b0fc477bb5551abb6da303d53ddb9908f2f6ce53be1920b2a09fc067"}
Sep 17 18:10:16.471711 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.471690    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:10:16.472590 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.472547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerStarted","Data":"8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792"}
Sep 17 18:10:16.472668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.472622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerStarted","Data":"449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a"}
Sep 17 18:10:16.472949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.472930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:16.472999 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.472989    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:16.474205 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.474181    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:10:16.518871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:16.518833    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podStartSLOduration=2.630911509 podStartE2EDuration="6.518820855s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.887909346s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:16.516109273 +0000 UTC m=+803.359272420" watchObservedRunningTime="2026-09-17 18:10:16.518820855 +0000 UTC m=+803.361984028"
Sep 17 18:10:17.477972 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:17.477926    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:10:22.483368 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:22.483338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:10:22.483955 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:22.483928    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:10:23.378259 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:23.378218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:24.384848 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:24.384803    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:32.484623 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:32.484571    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:10:33.378585 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:33.378528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:34.384514 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:34.384477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:38.597706 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:38.597669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerStarted","Data":"b9266eb61563b4bbb0bac445ce2d9e383940bbf9298964e9fe570d61272b7ce7"}
Sep 17 18:10:38.598118 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:38.597729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerStarted","Data":"d04b58da73426595a3ed2d965b3e1e9cc1c813d4fb344ceabb34ce92c827e405"}
Sep 17 18:10:38.598118 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:38.597959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:38.621578 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:38.621464    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podStartSLOduration=3.156579696 podStartE2EDuration="28.621443434s" totalImagesPullingTime="21.889543713s" totalInitContainerRuntime="3.575320025s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:38.620745632 +0000 UTC m=+825.463908841" watchObservedRunningTime="2026-09-17 18:10:38.621443434 +0000 UTC m=+825.464606583"
Sep 17 18:10:39.601890 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:39.601860    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:39.603005 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:39.602976    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:10:40.606428 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:40.606384    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:10:42.484697 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:42.484662    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:10:43.378464 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:43.378419    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:44.385056 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:44.385016    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:45.610934 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:45.610908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:10:45.611389 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:45.611365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:10:52.484554 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:52.484514    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:10:53.378133 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:53.378088    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:54.384846 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:54.384806    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:10:55.611972 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:10:55.611934    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:02.484713 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:02.484669    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:03.378047 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:03.378008    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:11:04.385227 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:04.385185    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:11:05.611791 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:05.611757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:10.579971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:10.579927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:11:12.483873 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:12.483836    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:14.384918 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:14.384873    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:11:15.612051 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:15.612016    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:20.581376 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:20.581344    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:11:22.484408 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.484365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:22.580469 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.580438    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:11:22.698378 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.698346    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"]
Sep 17 18:11:22.698606 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.698552    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container" containerID="cri-o://8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af" gracePeriod=30
Sep 17 18:11:22.698681 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.698613    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kube-rbac-proxy" containerID="cri-o://56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d" gracePeriod=30
Sep 17 18:11:22.819725 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.819698    2577 generic.go:356] "Generic (PLEG): container finished" podID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerID="56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d" exitCode=2
Sep 17 18:11:22.819859 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:22.819777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerDied","Data":"56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d"}
Sep 17 18:11:23.374289 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:23.374250    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 18:11:25.611459 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:25.611423    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:27.646054 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.646029    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:11:27.757949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.757860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls\") pod \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") "
Sep 17 18:11:27.757949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.757927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-cabundle-cert\") pod \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") "
Sep 17 18:11:27.758152 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.757960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8xpj\" (UniqueName: \"kubernetes.io/projected/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-api-access-f8xpj\") pod \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") "
Sep 17 18:11:27.758152 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.757990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-rbac-proxy-sar-config\") pod \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") "
Sep 17 18:11:27.758152 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.758011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kserve-provision-location\") pod \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\" (UID: \"89c63f14-ac92-4725-a4fe-3cbaaea071d2\") "
Sep 17 18:11:27.758401 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.758372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89c63f14-ac92-4725-a4fe-3cbaaea071d2" (UID: "89c63f14-ac92-4725-a4fe-3cbaaea071d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:27.758401 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.758377    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89c63f14-ac92-4725-a4fe-3cbaaea071d2" (UID: "89c63f14-ac92-4725-a4fe-3cbaaea071d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:27.758552 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.758376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89c63f14-ac92-4725-a4fe-3cbaaea071d2" (UID: "89c63f14-ac92-4725-a4fe-3cbaaea071d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:27.760127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.760103    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89c63f14-ac92-4725-a4fe-3cbaaea071d2" (UID: "89c63f14-ac92-4725-a4fe-3cbaaea071d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:27.760231 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.760153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-api-access-f8xpj" (OuterVolumeSpecName: "kube-api-access-f8xpj") pod "89c63f14-ac92-4725-a4fe-3cbaaea071d2" (UID: "89c63f14-ac92-4725-a4fe-3cbaaea071d2"). InnerVolumeSpecName "kube-api-access-f8xpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:27.850826 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.850787    2577 generic.go:356] "Generic (PLEG): container finished" podID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerID="8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af" exitCode=0
Sep 17 18:11:27.850978 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.850860    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"
Sep 17 18:11:27.850978 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.850868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerDied","Data":"8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af"}
Sep 17 18:11:27.850978 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.850929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6" event={"ID":"89c63f14-ac92-4725-a4fe-3cbaaea071d2","Type":"ContainerDied","Data":"d2e707954642924fb808c0ad84ebe6167a69c5d710813b6f4c3cdbec90326665"}
Sep 17 18:11:27.850978 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.850956    2577 scope.go:117] "RemoveContainer" containerID="56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d"
Sep 17 18:11:27.858647 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.858627    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:27.858761 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.858650    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:27.858761 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.858663    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c63f14-ac92-4725-a4fe-3cbaaea071d2-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:27.858761 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.858677    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c63f14-ac92-4725-a4fe-3cbaaea071d2-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:27.858761 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.858689    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8xpj\" (UniqueName: \"kubernetes.io/projected/89c63f14-ac92-4725-a4fe-3cbaaea071d2-kube-api-access-f8xpj\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:27.860770 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.860754    2577 scope.go:117] "RemoveContainer" containerID="8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af"
Sep 17 18:11:27.868273 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.868258    2577 scope.go:117] "RemoveContainer" containerID="f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0"
Sep 17 18:11:27.874930 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.874907    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"]
Sep 17 18:11:27.876343 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.876327    2577 scope.go:117] "RemoveContainer" containerID="56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d"
Sep 17 18:11:27.876627 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:11:27.876611    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d\": container with ID starting with 56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d not found: ID does not exist" containerID="56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d"
Sep 17 18:11:27.876683 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.876634    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d"} err="failed to get container status \"56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d\": rpc error: code = NotFound desc = could not find container \"56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d\": container with ID starting with 56292d30b04cc697d65dfbfd839e32291f167b9157220881ba3ce05d4fbc0b2d not found: ID does not exist"
Sep 17 18:11:27.876683 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.876649    2577 scope.go:117] "RemoveContainer" containerID="8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af"
Sep 17 18:11:27.876902 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:11:27.876876    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af\": container with ID starting with 8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af not found: ID does not exist" containerID="8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af"
Sep 17 18:11:27.876969 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.876910    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af"} err="failed to get container status \"8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af\": rpc error: code = NotFound desc = could not find container \"8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af\": container with ID starting with 8d0a2799c8c5b6daa10fd40addb159a178b45bb309e1c096ebc724d20ddc39af not found: ID does not exist"
Sep 17 18:11:27.876969 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.876934    2577 scope.go:117] "RemoveContainer" containerID="f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0"
Sep 17 18:11:27.877184 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:11:27.877169    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0\": container with ID starting with f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0 not found: ID does not exist" containerID="f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0"
Sep 17 18:11:27.877231 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.877187    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0"} err="failed to get container status \"f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0\": rpc error: code = NotFound desc = could not find container \"f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0\": container with ID starting with f6c7f6f9b7f592524877e57a79487e005d31b7b8da918ab456dd7ffc944be9e0 not found: ID does not exist"
Sep 17 18:11:27.880190 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:27.880168    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-b47j6"]
Sep 17 18:11:29.585213 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:29.585183    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" path="/var/lib/kubelet/pods/89c63f14-ac92-4725-a4fe-3cbaaea071d2/volumes"
Sep 17 18:11:30.516316 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:30.516282    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"]
Sep 17 18:11:30.516661 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:30.516632    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" containerID="cri-o://a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a" gracePeriod=30
Sep 17 18:11:30.516798 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:30.516697    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kube-rbac-proxy" containerID="cri-o://af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd" gracePeriod=30
Sep 17 18:11:30.869420 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:30.869328    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d63981d-842e-4d90-9564-c44236b6566d" containerID="af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd" exitCode=2
Sep 17 18:11:30.869420 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:30.869375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerDied","Data":"af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd"}
Sep 17 18:11:32.484198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:32.484161    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:32.580643 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:32.580602    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:11:34.379398 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:34.379355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:11:35.611991 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.611954    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:35.880108 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.880085    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:11:35.895153 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.895115    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d63981d-842e-4d90-9564-c44236b6566d" containerID="a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a" exitCode=0
Sep 17 18:11:35.895291 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.895170    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"
Sep 17 18:11:35.895291 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.895188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerDied","Data":"a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a"}
Sep 17 18:11:35.895291 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.895245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f" event={"ID":"9d63981d-842e-4d90-9564-c44236b6566d","Type":"ContainerDied","Data":"48df575966b9d403a2ab8f50de0cfa273a16b6812faea46118acb620a8872506"}
Sep 17 18:11:35.895291 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.895271    2577 scope.go:117] "RemoveContainer" containerID="af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd"
Sep 17 18:11:35.903993 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.903974    2577 scope.go:117] "RemoveContainer" containerID="a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a"
Sep 17 18:11:35.912956 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.912934    2577 scope.go:117] "RemoveContainer" containerID="cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb"
Sep 17 18:11:35.920900 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.920878    2577 scope.go:117] "RemoveContainer" containerID="af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd"
Sep 17 18:11:35.921207 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:11:35.921182    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd\": container with ID starting with af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd not found: ID does not exist" containerID="af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd"
Sep 17 18:11:35.921259 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.921222    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd"} err="failed to get container status \"af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd\": rpc error: code = NotFound desc = could not find container \"af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd\": container with ID starting with af17b4269513a0d17af8ce1778e432f043d6ea23953a64d69fb0b829a289e7dd not found: ID does not exist"
Sep 17 18:11:35.921259 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.921248    2577 scope.go:117] "RemoveContainer" containerID="a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a"
Sep 17 18:11:35.921505 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:11:35.921484    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a\": container with ID starting with a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a not found: ID does not exist" containerID="a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a"
Sep 17 18:11:35.921550 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.921512    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a"} err="failed to get container status \"a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a\": rpc error: code = NotFound desc = could not find container \"a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a\": container with ID starting with a2f31ac7170cd156b37118e2984c495d7eb1851b6c62c234e6264df5b72dc81a not found: ID does not exist"
Sep 17 18:11:35.921550 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.921533    2577 scope.go:117] "RemoveContainer" containerID="cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb"
Sep 17 18:11:35.921827 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:11:35.921808    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb\": container with ID starting with cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb not found: ID does not exist" containerID="cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb"
Sep 17 18:11:35.921882 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:35.921834    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb"} err="failed to get container status \"cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb\": rpc error: code = NotFound desc = could not find container \"cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb\": container with ID starting with cfb8a32fbbe913055c64cc79d30b9ce682d40caf1e4af39c42a614a4134f8cfb not found: ID does not exist"
Sep 17 18:11:36.031423 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-cabundle-cert\") pod \"9d63981d-842e-4d90-9564-c44236b6566d\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") "
Sep 17 18:11:36.031423 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-kube-rbac-proxy-sar-config\") pod \"9d63981d-842e-4d90-9564-c44236b6566d\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") "
Sep 17 18:11:36.031722 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d63981d-842e-4d90-9564-c44236b6566d-kserve-provision-location\") pod \"9d63981d-842e-4d90-9564-c44236b6566d\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") "
Sep 17 18:11:36.031722 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c8zx\" (UniqueName: \"kubernetes.io/projected/9d63981d-842e-4d90-9564-c44236b6566d-kube-api-access-7c8zx\") pod \"9d63981d-842e-4d90-9564-c44236b6566d\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") "
Sep 17 18:11:36.031722 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls\") pod \"9d63981d-842e-4d90-9564-c44236b6566d\" (UID: \"9d63981d-842e-4d90-9564-c44236b6566d\") "
Sep 17 18:11:36.031855 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d63981d-842e-4d90-9564-c44236b6566d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d63981d-842e-4d90-9564-c44236b6566d" (UID: "9d63981d-842e-4d90-9564-c44236b6566d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:36.031910 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d63981d-842e-4d90-9564-c44236b6566d" (UID: "9d63981d-842e-4d90-9564-c44236b6566d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:36.031910 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d63981d-842e-4d90-9564-c44236b6566d" (UID: "9d63981d-842e-4d90-9564-c44236b6566d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:36.031995 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.031923    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d63981d-842e-4d90-9564-c44236b6566d-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:36.033669 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.033645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d63981d-842e-4d90-9564-c44236b6566d" (UID: "9d63981d-842e-4d90-9564-c44236b6566d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:36.033796 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.033669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d63981d-842e-4d90-9564-c44236b6566d-kube-api-access-7c8zx" (OuterVolumeSpecName: "kube-api-access-7c8zx") pod "9d63981d-842e-4d90-9564-c44236b6566d" (UID: "9d63981d-842e-4d90-9564-c44236b6566d"). InnerVolumeSpecName "kube-api-access-7c8zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:36.133143 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.133017    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:36.133143 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.133062    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d63981d-842e-4d90-9564-c44236b6566d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:36.133143 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.133073    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8zx\" (UniqueName: \"kubernetes.io/projected/9d63981d-842e-4d90-9564-c44236b6566d-kube-api-access-7c8zx\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:36.133143 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.133084    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d63981d-842e-4d90-9564-c44236b6566d-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:11:36.218070 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.218039    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"]
Sep 17 18:11:36.221434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:36.221411    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-wfs4f"]
Sep 17 18:11:37.584815 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:37.584780    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d63981d-842e-4d90-9564-c44236b6566d" path="/var/lib/kubelet/pods/9d63981d-842e-4d90-9564-c44236b6566d/volumes"
Sep 17 18:11:42.485298 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:42.485267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:11:45.611619 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:45.611554    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:55.585165 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:55.585139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:11:55.798683 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:55.798659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:11:55.799881 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:11:55.799863    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:12:10.974859 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:10.974825    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"]
Sep 17 18:12:10.975342 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:10.975090    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kube-rbac-proxy" containerID="cri-o://8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792" gracePeriod=30
Sep 17 18:12:10.975342 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:10.975077    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" containerID="cri-o://449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a" gracePeriod=30
Sep 17 18:12:11.069908 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.069880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"]
Sep 17 18:12:11.070152 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.070130    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kube-rbac-proxy" containerID="cri-o://b9266eb61563b4bbb0bac445ce2d9e383940bbf9298964e9fe570d61272b7ce7" gracePeriod=30
Sep 17 18:12:11.070250 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.070110    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container" containerID="cri-o://d04b58da73426595a3ed2d965b3e1e9cc1c813d4fb344ceabb34ce92c827e405" gracePeriod=30
Sep 17 18:12:11.145328 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145297    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"]
Sep 17 18:12:11.145693 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145680    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kube-rbac-proxy"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145695    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kube-rbac-proxy"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145707    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="storage-initializer"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145713    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="storage-initializer"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="storage-initializer"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145731    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="storage-initializer"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145738    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container"
Sep 17 18:12:11.145744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145743    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145758    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kube-rbac-proxy"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145763    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kube-rbac-proxy"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145772    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145777    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145827    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kube-rbac-proxy"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145834    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kserve-container"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145843    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d63981d-842e-4d90-9564-c44236b6566d" containerName="kube-rbac-proxy"
Sep 17 18:12:11.145961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.145851    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c63f14-ac92-4725-a4fe-3cbaaea071d2" containerName="kserve-container"
Sep 17 18:12:11.148024 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.148003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.156127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.155946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-fc652-kube-rbac-proxy-sar-config\""
Sep 17 18:12:11.156127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.156079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-fc652-predictor-serving-cert\""
Sep 17 18:12:11.160002 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.159980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"]
Sep 17 18:12:11.217290 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.217262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ltd\" (UniqueName: \"kubernetes.io/projected/d5332680-2e45-4515-8f92-ed7c03120327-kube-api-access-x7ltd\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.217416 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.217302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.217416 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.217321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5332680-2e45-4515-8f92-ed7c03120327-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.318472 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.318387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ltd\" (UniqueName: \"kubernetes.io/projected/d5332680-2e45-4515-8f92-ed7c03120327-kube-api-access-x7ltd\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.318472 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.318433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.318472 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.318451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5332680-2e45-4515-8f92-ed7c03120327-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.318759 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:11.318547    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-serving-cert: secret "message-dumper-raw-fc652-predictor-serving-cert" not found
Sep 17 18:12:11.318759 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:11.318649    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls podName:d5332680-2e45-4515-8f92-ed7c03120327 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:11.818629257 +0000 UTC m=+918.661792400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls") pod "message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" (UID: "d5332680-2e45-4515-8f92-ed7c03120327") : secret "message-dumper-raw-fc652-predictor-serving-cert" not found
Sep 17 18:12:11.319139 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.319123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5332680-2e45-4515-8f92-ed7c03120327-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.328986 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.328960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ltd\" (UniqueName: \"kubernetes.io/projected/d5332680-2e45-4515-8f92-ed7c03120327-kube-api-access-x7ltd\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.823586 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.823530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:11.825895 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:11.825866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls\") pod \"message-dumper-raw-fc652-predictor-6d55ffc99b-44n82\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") " pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:12.057797 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.057761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:12.079959 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.079898    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f2fc54b-3b28-4f25-a085-766376c72102" containerID="b9266eb61563b4bbb0bac445ce2d9e383940bbf9298964e9fe570d61272b7ce7" exitCode=2
Sep 17 18:12:12.080077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.079966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerDied","Data":"b9266eb61563b4bbb0bac445ce2d9e383940bbf9298964e9fe570d61272b7ce7"}
Sep 17 18:12:12.082214 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.082192    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerID="8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792" exitCode=2
Sep 17 18:12:12.082311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.082231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerDied","Data":"8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792"}
Sep 17 18:12:12.326624 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.326500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"]
Sep 17 18:12:12.330968 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:12:12.330903    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5332680_2e45_4515_8f92_ed7c03120327.slice/crio-ec3f5f3ba8e22b6a4e2e55f5915cdf8a99f0186fde6929dc608d35e5d62ecb24 WatchSource:0}: Error finding container ec3f5f3ba8e22b6a4e2e55f5915cdf8a99f0186fde6929dc608d35e5d62ecb24: Status 404 returned error can't find the container with id ec3f5f3ba8e22b6a4e2e55f5915cdf8a99f0186fde6929dc608d35e5d62ecb24
Sep 17 18:12:12.478505 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.478460    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 18:12:12.484105 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:12.484078    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:12:13.089027 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:13.088993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" event={"ID":"d5332680-2e45-4515-8f92-ed7c03120327","Type":"ContainerStarted","Data":"ec3f5f3ba8e22b6a4e2e55f5915cdf8a99f0186fde6929dc608d35e5d62ecb24"}
Sep 17 18:12:14.101948 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:14.101908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" event={"ID":"d5332680-2e45-4515-8f92-ed7c03120327","Type":"ContainerStarted","Data":"a0ad717505e13f7de111fad5b4c46bc626da52e3b9592734ad9497cb76290f15"}
Sep 17 18:12:14.102397 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:14.102065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:14.102397 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:14.102099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" event={"ID":"d5332680-2e45-4515-8f92-ed7c03120327","Type":"ContainerStarted","Data":"80cffd8a20842be6b5e52dc27f8a863368efd6ec3ae20251bc8ec6117266336b"}
Sep 17 18:12:14.102397 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:14.102124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:14.104096 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:14.104072    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:14.120764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:14.120723    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" podStartSLOduration=2.018951081 podStartE2EDuration="3.120712196s" totalImagesPullingTime="1.101761115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:14.120462144 +0000 UTC m=+920.963625292" watchObservedRunningTime="2026-09-17 18:12:14.120712196 +0000 UTC m=+920.963875343"
Sep 17 18:12:15.109452 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.109416    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f2fc54b-3b28-4f25-a085-766376c72102" containerID="d04b58da73426595a3ed2d965b3e1e9cc1c813d4fb344ceabb34ce92c827e405" exitCode=0
Sep 17 18:12:15.109935 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.109478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerDied","Data":"d04b58da73426595a3ed2d965b3e1e9cc1c813d4fb344ceabb34ce92c827e405"}
Sep 17 18:12:15.219010 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.218987    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:12:15.251307 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5czfn\" (UniqueName: \"kubernetes.io/projected/5f2fc54b-3b28-4f25-a085-766376c72102-kube-api-access-5czfn\") pod \"5f2fc54b-3b28-4f25-a085-766376c72102\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") "
Sep 17 18:12:15.251460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls\") pod \"5f2fc54b-3b28-4f25-a085-766376c72102\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") "
Sep 17 18:12:15.251460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-kube-rbac-proxy-sar-config\") pod \"5f2fc54b-3b28-4f25-a085-766376c72102\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") "
Sep 17 18:12:15.251460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-cabundle-cert\") pod \"5f2fc54b-3b28-4f25-a085-766376c72102\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") "
Sep 17 18:12:15.251460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251452    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2fc54b-3b28-4f25-a085-766376c72102-kserve-provision-location\") pod \"5f2fc54b-3b28-4f25-a085-766376c72102\" (UID: \"5f2fc54b-3b28-4f25-a085-766376c72102\") "
Sep 17 18:12:15.251829 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f2fc54b-3b28-4f25-a085-766376c72102" (UID: "5f2fc54b-3b28-4f25-a085-766376c72102"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:15.251908 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.251882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2fc54b-3b28-4f25-a085-766376c72102-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f2fc54b-3b28-4f25-a085-766376c72102" (UID: "5f2fc54b-3b28-4f25-a085-766376c72102"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:15.252045 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.252024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f2fc54b-3b28-4f25-a085-766376c72102" (UID: "5f2fc54b-3b28-4f25-a085-766376c72102"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:15.253526 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.253493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2fc54b-3b28-4f25-a085-766376c72102-kube-api-access-5czfn" (OuterVolumeSpecName: "kube-api-access-5czfn") pod "5f2fc54b-3b28-4f25-a085-766376c72102" (UID: "5f2fc54b-3b28-4f25-a085-766376c72102"). InnerVolumeSpecName "kube-api-access-5czfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:15.253526 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.253496    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f2fc54b-3b28-4f25-a085-766376c72102" (UID: "5f2fc54b-3b28-4f25-a085-766376c72102"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:15.352246 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.352156    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czfn\" (UniqueName: \"kubernetes.io/projected/5f2fc54b-3b28-4f25-a085-766376c72102-kube-api-access-5czfn\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.352246 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.352181    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2fc54b-3b28-4f25-a085-766376c72102-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.352246 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.352192    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.352246 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.352202    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2fc54b-3b28-4f25-a085-766376c72102-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.352246 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.352211    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2fc54b-3b28-4f25-a085-766376c72102-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.707943 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.707920    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:12:15.754919 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.754889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb1a5a2-6151-4bdc-9568-002c61edb068-proxy-tls\") pod \"ebb1a5a2-6151-4bdc-9568-002c61edb068\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") "
Sep 17 18:12:15.755086 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-cabundle-cert\") pod \"ebb1a5a2-6151-4bdc-9568-002c61edb068\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") "
Sep 17 18:12:15.755156 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb1a5a2-6151-4bdc-9568-002c61edb068-kserve-provision-location\") pod \"ebb1a5a2-6151-4bdc-9568-002c61edb068\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") "
Sep 17 18:12:15.755156 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wc8b\" (UniqueName: \"kubernetes.io/projected/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-api-access-5wc8b\") pod \"ebb1a5a2-6151-4bdc-9568-002c61edb068\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") "
Sep 17 18:12:15.755233 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-rbac-proxy-sar-config\") pod \"ebb1a5a2-6151-4bdc-9568-002c61edb068\" (UID: \"ebb1a5a2-6151-4bdc-9568-002c61edb068\") "
Sep 17 18:12:15.755473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb1a5a2-6151-4bdc-9568-002c61edb068-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebb1a5a2-6151-4bdc-9568-002c61edb068" (UID: "ebb1a5a2-6151-4bdc-9568-002c61edb068"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:15.755624 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebb1a5a2-6151-4bdc-9568-002c61edb068" (UID: "ebb1a5a2-6151-4bdc-9568-002c61edb068"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:15.755624 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.755600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebb1a5a2-6151-4bdc-9568-002c61edb068" (UID: "ebb1a5a2-6151-4bdc-9568-002c61edb068"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:15.756894 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.756874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb1a5a2-6151-4bdc-9568-002c61edb068-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebb1a5a2-6151-4bdc-9568-002c61edb068" (UID: "ebb1a5a2-6151-4bdc-9568-002c61edb068"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:15.757092 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.757074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-api-access-5wc8b" (OuterVolumeSpecName: "kube-api-access-5wc8b") pod "ebb1a5a2-6151-4bdc-9568-002c61edb068" (UID: "ebb1a5a2-6151-4bdc-9568-002c61edb068"). InnerVolumeSpecName "kube-api-access-5wc8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:15.856395 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.856365    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.856395 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.856391    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb1a5a2-6151-4bdc-9568-002c61edb068-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.856395 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.856400    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wc8b\" (UniqueName: \"kubernetes.io/projected/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-api-access-5wc8b\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.856660 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.856408    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb1a5a2-6151-4bdc-9568-002c61edb068-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.856660 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:15.856419    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb1a5a2-6151-4bdc-9568-002c61edb068-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:12:16.116997 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.116967    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"
Sep 17 18:12:16.117448 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.116967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt" event={"ID":"5f2fc54b-3b28-4f25-a085-766376c72102","Type":"ContainerDied","Data":"b4c32a850e58042ed31f848751a2ef3d8ae3f84827d28174593668e7ea3dac31"}
Sep 17 18:12:16.117448 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.117128    2577 scope.go:117] "RemoveContainer" containerID="b9266eb61563b4bbb0bac445ce2d9e383940bbf9298964e9fe570d61272b7ce7"
Sep 17 18:12:16.119493 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.119468    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerID="449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a" exitCode=0
Sep 17 18:12:16.119662 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.119525    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"
Sep 17 18:12:16.119662 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.119579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerDied","Data":"449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a"}
Sep 17 18:12:16.119662 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.119640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd" event={"ID":"ebb1a5a2-6151-4bdc-9568-002c61edb068","Type":"ContainerDied","Data":"c284c3dc1fc48a04dc2027c2c32433f9c284c8f6706c731ea0ff9213a5aece26"}
Sep 17 18:12:16.126797 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.126775    2577 scope.go:117] "RemoveContainer" containerID="d04b58da73426595a3ed2d965b3e1e9cc1c813d4fb344ceabb34ce92c827e405"
Sep 17 18:12:16.134356 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.134339    2577 scope.go:117] "RemoveContainer" containerID="af8d9251b0fc477bb5551abb6da303d53ddb9908f2f6ce53be1920b2a09fc067"
Sep 17 18:12:16.136333 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.136315    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"]
Sep 17 18:12:16.139265 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.139246    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7da3c-predictor-ffb8dcf4f-jppwt"]
Sep 17 18:12:16.142490 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.142474    2577 scope.go:117] "RemoveContainer" containerID="8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792"
Sep 17 18:12:16.149573 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.149546    2577 scope.go:117] "RemoveContainer" containerID="449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a"
Sep 17 18:12:16.154130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.154108    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"]
Sep 17 18:12:16.156173 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.156153    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7da3c-predictor-6d49d49664-gktvd"]
Sep 17 18:12:16.158311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.158297    2577 scope.go:117] "RemoveContainer" containerID="1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a"
Sep 17 18:12:16.167662 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.167642    2577 scope.go:117] "RemoveContainer" containerID="8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792"
Sep 17 18:12:16.167972 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:16.167952    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792\": container with ID starting with 8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792 not found: ID does not exist" containerID="8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792"
Sep 17 18:12:16.168023 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.167980    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792"} err="failed to get container status \"8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792\": rpc error: code = NotFound desc = could not find container \"8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792\": container with ID starting with 8ecd34741d5c9f80967e933df9fcbcf349c834bde1bd4b4d8da6cea823e88792 not found: ID does not exist"
Sep 17 18:12:16.168023 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.167997    2577 scope.go:117] "RemoveContainer" containerID="449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a"
Sep 17 18:12:16.168239 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:16.168225    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a\": container with ID starting with 449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a not found: ID does not exist" containerID="449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a"
Sep 17 18:12:16.168283 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.168244    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a"} err="failed to get container status \"449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a\": rpc error: code = NotFound desc = could not find container \"449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a\": container with ID starting with 449bff7a73a148cf8241495e206390e6f5d6236172bd1afbdc149d2392db287a not found: ID does not exist"
Sep 17 18:12:16.168283 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.168256    2577 scope.go:117] "RemoveContainer" containerID="1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a"
Sep 17 18:12:16.168517 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:16.168499    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a\": container with ID starting with 1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a not found: ID does not exist" containerID="1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a"
Sep 17 18:12:16.168583 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:16.168521    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a"} err="failed to get container status \"1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a\": rpc error: code = NotFound desc = could not find container \"1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a\": container with ID starting with 1a0d944e44a59b45e7c75f31f15f76e3b35782e2933e55be4c3eb48bbb79470a not found: ID does not exist"
Sep 17 18:12:17.585432 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:17.585400    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" path="/var/lib/kubelet/pods/5f2fc54b-3b28-4f25-a085-766376c72102/volumes"
Sep 17 18:12:17.585920 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:17.585905    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" path="/var/lib/kubelet/pods/ebb1a5a2-6151-4bdc-9568-002c61edb068/volumes"
Sep 17 18:12:21.124509 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:21.124476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:12:31.035730 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.035695    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"]
Sep 17 18:12:31.036121 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036100    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="storage-initializer"
Sep 17 18:12:31.036121 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036113    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="storage-initializer"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036129    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="storage-initializer"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036137    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="storage-initializer"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036143    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036149    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036163    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kube-rbac-proxy"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kube-rbac-proxy"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036176    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kube-rbac-proxy"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036182    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kube-rbac-proxy"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036190    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container"
Sep 17 18:12:31.036198 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container"
Sep 17 18:12:31.036497 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036253    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kserve-container"
Sep 17 18:12:31.036497 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036262    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kube-rbac-proxy"
Sep 17 18:12:31.036497 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036268    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb1a5a2-6151-4bdc-9568-002c61edb068" containerName="kube-rbac-proxy"
Sep 17 18:12:31.036497 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.036277    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2fc54b-3b28-4f25-a085-766376c72102" containerName="kserve-container"
Sep 17 18:12:31.040388 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.040374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.043057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.043032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-fc652-predictor-serving-cert\""
Sep 17 18:12:31.043196 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.043077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:12:31.043196 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.043090    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:12:31.043196 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.043177    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-fc652-kube-rbac-proxy-sar-config\""
Sep 17 18:12:31.049391 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.049369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"]
Sep 17 18:12:31.081046 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.081022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-cabundle-cert\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.081211 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.081075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.081211 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.081151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpwf\" (UniqueName: \"kubernetes.io/projected/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-api-access-clpwf\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.081211 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.081203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8046de1-3a6d-458f-a741-7637c42dfa48-kserve-provision-location\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.081357 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.081234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182035 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.181998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpwf\" (UniqueName: \"kubernetes.io/projected/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-api-access-clpwf\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8046de1-3a6d-458f-a741-7637c42dfa48-kserve-provision-location\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-cabundle-cert\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182462 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:31.182236    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-serving-cert: secret "isvc-logger-raw-fc652-predictor-serving-cert" not found
Sep 17 18:12:31.182462 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:12:31.182317    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls podName:e8046de1-3a6d-458f-a741-7637c42dfa48 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:31.682295773 +0000 UTC m=+938.525458900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls") pod "isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" (UID: "e8046de1-3a6d-458f-a741-7637c42dfa48") : secret "isvc-logger-raw-fc652-predictor-serving-cert" not found
Sep 17 18:12:31.182615 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8046de1-3a6d-458f-a741-7637c42dfa48-kserve-provision-location\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182900 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.182900 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.182895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-cabundle-cert\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.192698 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.192677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpwf\" (UniqueName: \"kubernetes.io/projected/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-api-access-clpwf\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.687486 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.687451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.690011 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.689979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls\") pod \"isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") " pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:31.950955 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:31.950859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:32.227195 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:32.227169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"]
Sep 17 18:12:32.228906 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:12:32.228881    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8046de1_3a6d_458f_a741_7637c42dfa48.slice/crio-dfeb72125a5070f17552b66e20e107fe58dacba8e2145a0e96980a06d7177155 WatchSource:0}: Error finding container dfeb72125a5070f17552b66e20e107fe58dacba8e2145a0e96980a06d7177155: Status 404 returned error can't find the container with id dfeb72125a5070f17552b66e20e107fe58dacba8e2145a0e96980a06d7177155
Sep 17 18:12:33.200521 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:33.200489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerStarted","Data":"bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d"}
Sep 17 18:12:33.200727 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:33.200556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerStarted","Data":"dfeb72125a5070f17552b66e20e107fe58dacba8e2145a0e96980a06d7177155"}
Sep 17 18:12:37.221317 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:37.221287    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerID="bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d" exitCode=0
Sep 17 18:12:37.221709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:37.221361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerDied","Data":"bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d"}
Sep 17 18:12:38.228775 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.228739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerStarted","Data":"082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187"}
Sep 17 18:12:38.229218 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.228805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerStarted","Data":"2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d"}
Sep 17 18:12:38.229218 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.228828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerStarted","Data":"79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e"}
Sep 17 18:12:38.229218 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.229124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:38.229218 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.229190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:38.229377 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.229226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:38.230712 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.230678    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:12:38.231293 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.231274    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:38.251406 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:38.251356    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podStartSLOduration=3.247329975 podStartE2EDuration="7.251341927s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.004011952s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:38.249077069 +0000 UTC m=+945.092240217" watchObservedRunningTime="2026-09-17 18:12:38.251341927 +0000 UTC m=+945.094505075"
Sep 17 18:12:39.233777 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:39.233742    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:12:39.234173 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:39.234085    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:44.237765 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:44.237737    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:12:44.238264 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:44.238235    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:12:44.238674 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:44.238654    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:54.239129 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:54.239088    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:12:54.239658 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:12:54.239539    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:04.238376 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:04.238336    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:13:04.238886 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:04.238752    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:14.238388 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:14.238349    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:13:14.238927 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:14.238789    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:24.238293 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:24.238250    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:13:24.238978 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:24.238673    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:34.238171 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:34.238131    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:13:34.238656 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:34.238588    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:44.238234 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:44.238188    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:13:44.238766 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:44.238488    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:48.580259 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:48.580221    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:13:48.580723 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:48.580520    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:13:58.581359 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:58.581326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:13:58.581843 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:13:58.581823    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:14:06.287275 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.287236    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-fc652-predictor-6d55ffc99b-44n82_d5332680-2e45-4515-8f92-ed7c03120327/kserve-container/0.log"
Sep 17 18:14:06.442130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.442100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"]
Sep 17 18:14:06.442470 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.442437    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" containerID="cri-o://79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e" gracePeriod=30
Sep 17 18:14:06.442588 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.442461    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" containerID="cri-o://2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d" gracePeriod=30
Sep 17 18:14:06.442588 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.442450    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" containerID="cri-o://082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187" gracePeriod=30
Sep 17 18:14:06.460447 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.460419    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"]
Sep 17 18:14:06.460750 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.460704    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kserve-container" containerID="cri-o://80cffd8a20842be6b5e52dc27f8a863368efd6ec3ae20251bc8ec6117266336b" gracePeriod=30
Sep 17 18:14:06.460855 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.460758    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kube-rbac-proxy" containerID="cri-o://a0ad717505e13f7de111fad5b4c46bc626da52e3b9592734ad9497cb76290f15" gracePeriod=30
Sep 17 18:14:06.662111 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.662080    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerID="2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d" exitCode=2
Sep 17 18:14:06.662281 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.662158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerDied","Data":"2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d"}
Sep 17 18:14:06.664112 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.664091    2577 generic.go:356] "Generic (PLEG): container finished" podID="d5332680-2e45-4515-8f92-ed7c03120327" containerID="a0ad717505e13f7de111fad5b4c46bc626da52e3b9592734ad9497cb76290f15" exitCode=2
Sep 17 18:14:06.664112 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.664107    2577 generic.go:356] "Generic (PLEG): container finished" podID="d5332680-2e45-4515-8f92-ed7c03120327" containerID="80cffd8a20842be6b5e52dc27f8a863368efd6ec3ae20251bc8ec6117266336b" exitCode=2
Sep 17 18:14:06.664252 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.664147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" event={"ID":"d5332680-2e45-4515-8f92-ed7c03120327","Type":"ContainerDied","Data":"a0ad717505e13f7de111fad5b4c46bc626da52e3b9592734ad9497cb76290f15"}
Sep 17 18:14:06.664252 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.664195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" event={"ID":"d5332680-2e45-4515-8f92-ed7c03120327","Type":"ContainerDied","Data":"80cffd8a20842be6b5e52dc27f8a863368efd6ec3ae20251bc8ec6117266336b"}
Sep 17 18:14:06.713374 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.713347    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:14:06.885208 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.885118    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5332680-2e45-4515-8f92-ed7c03120327-kube-rbac-proxy-sar-config\") pod \"d5332680-2e45-4515-8f92-ed7c03120327\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") "
Sep 17 18:14:06.885359 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.885214    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls\") pod \"d5332680-2e45-4515-8f92-ed7c03120327\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") "
Sep 17 18:14:06.885359 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.885258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7ltd\" (UniqueName: \"kubernetes.io/projected/d5332680-2e45-4515-8f92-ed7c03120327-kube-api-access-x7ltd\") pod \"d5332680-2e45-4515-8f92-ed7c03120327\" (UID: \"d5332680-2e45-4515-8f92-ed7c03120327\") "
Sep 17 18:14:06.885537 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.885509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5332680-2e45-4515-8f92-ed7c03120327-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5332680-2e45-4515-8f92-ed7c03120327" (UID: "d5332680-2e45-4515-8f92-ed7c03120327"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:06.887258 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.887235    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5332680-2e45-4515-8f92-ed7c03120327" (UID: "d5332680-2e45-4515-8f92-ed7c03120327"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:06.887312 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.887269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5332680-2e45-4515-8f92-ed7c03120327-kube-api-access-x7ltd" (OuterVolumeSpecName: "kube-api-access-x7ltd") pod "d5332680-2e45-4515-8f92-ed7c03120327" (UID: "d5332680-2e45-4515-8f92-ed7c03120327"). InnerVolumeSpecName "kube-api-access-x7ltd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:06.986586 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.986542    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5332680-2e45-4515-8f92-ed7c03120327-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:06.986586 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.986594    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5332680-2e45-4515-8f92-ed7c03120327-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:06.986866 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:06.986604    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ltd\" (UniqueName: \"kubernetes.io/projected/d5332680-2e45-4515-8f92-ed7c03120327-kube-api-access-x7ltd\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:07.670232 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:07.670197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82" event={"ID":"d5332680-2e45-4515-8f92-ed7c03120327","Type":"ContainerDied","Data":"ec3f5f3ba8e22b6a4e2e55f5915cdf8a99f0186fde6929dc608d35e5d62ecb24"}
Sep 17 18:14:07.670232 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:07.670226    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"
Sep 17 18:14:07.670722 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:07.670265    2577 scope.go:117] "RemoveContainer" containerID="a0ad717505e13f7de111fad5b4c46bc626da52e3b9592734ad9497cb76290f15"
Sep 17 18:14:07.678644 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:07.678619    2577 scope.go:117] "RemoveContainer" containerID="80cffd8a20842be6b5e52dc27f8a863368efd6ec3ae20251bc8ec6117266336b"
Sep 17 18:14:07.688837 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:07.688810    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"]
Sep 17 18:14:07.696614 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:07.694004    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fc652-predictor-6d55ffc99b-44n82"]
Sep 17 18:14:08.580991 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:08.580945    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:14:08.582210 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:08.582186    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:14:09.234083 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:09.234039    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:14:09.585008 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:09.584933    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5332680-2e45-4515-8f92-ed7c03120327" path="/var/lib/kubelet/pods/d5332680-2e45-4515-8f92-ed7c03120327/volumes"
Sep 17 18:14:11.692716 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:11.692682    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerID="79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e" exitCode=0
Sep 17 18:14:11.693097 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:11.692756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerDied","Data":"79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e"}
Sep 17 18:14:14.234357 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:14.234318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:14:18.581093 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:18.581052    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:14:18.582292 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:18.582273    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:14:19.234437 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:19.234397    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:14:19.234658 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:19.234586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:14:24.234006 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:24.233962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:14:28.580936 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:28.580888    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:14:28.581321 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:28.581030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:14:28.582120 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:28.582098    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:14:28.582204 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:28.582190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:14:29.234673 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:29.234633    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:14:34.234392 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:34.234339    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:14:36.622777 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.622752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:14:36.697668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.697555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8046de1-3a6d-458f-a741-7637c42dfa48-kserve-provision-location\") pod \"e8046de1-3a6d-458f-a741-7637c42dfa48\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") "
Sep 17 18:14:36.697668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.697619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-cabundle-cert\") pod \"e8046de1-3a6d-458f-a741-7637c42dfa48\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") "
Sep 17 18:14:36.697668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.697650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-rbac-proxy-sar-config\") pod \"e8046de1-3a6d-458f-a741-7637c42dfa48\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") "
Sep 17 18:14:36.697940 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.697676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clpwf\" (UniqueName: \"kubernetes.io/projected/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-api-access-clpwf\") pod \"e8046de1-3a6d-458f-a741-7637c42dfa48\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") "
Sep 17 18:14:36.697940 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.697707    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls\") pod \"e8046de1-3a6d-458f-a741-7637c42dfa48\" (UID: \"e8046de1-3a6d-458f-a741-7637c42dfa48\") "
Sep 17 18:14:36.697940 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.697890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8046de1-3a6d-458f-a741-7637c42dfa48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8046de1-3a6d-458f-a741-7637c42dfa48" (UID: "e8046de1-3a6d-458f-a741-7637c42dfa48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:36.698061 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.698011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8046de1-3a6d-458f-a741-7637c42dfa48" (UID: "e8046de1-3a6d-458f-a741-7637c42dfa48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:36.698061 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.698019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8046de1-3a6d-458f-a741-7637c42dfa48" (UID: "e8046de1-3a6d-458f-a741-7637c42dfa48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:36.699796 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.699776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-api-access-clpwf" (OuterVolumeSpecName: "kube-api-access-clpwf") pod "e8046de1-3a6d-458f-a741-7637c42dfa48" (UID: "e8046de1-3a6d-458f-a741-7637c42dfa48"). InnerVolumeSpecName "kube-api-access-clpwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:36.699923 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.699899    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8046de1-3a6d-458f-a741-7637c42dfa48" (UID: "e8046de1-3a6d-458f-a741-7637c42dfa48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:36.798502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.798450    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8046de1-3a6d-458f-a741-7637c42dfa48-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.798502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.798493    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.798502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.798503    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.798502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.798512    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpwf\" (UniqueName: \"kubernetes.io/projected/e8046de1-3a6d-458f-a741-7637c42dfa48-kube-api-access-clpwf\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.798502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.798524    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8046de1-3a6d-458f-a741-7637c42dfa48-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:14:36.807406 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.807377    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerID="082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187" exitCode=137
Sep 17 18:14:36.807539 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.807454    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"
Sep 17 18:14:36.807539 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.807464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerDied","Data":"082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187"}
Sep 17 18:14:36.807539 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.807532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd" event={"ID":"e8046de1-3a6d-458f-a741-7637c42dfa48","Type":"ContainerDied","Data":"dfeb72125a5070f17552b66e20e107fe58dacba8e2145a0e96980a06d7177155"}
Sep 17 18:14:36.807706 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.807579    2577 scope.go:117] "RemoveContainer" containerID="082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187"
Sep 17 18:14:36.816248 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.816227    2577 scope.go:117] "RemoveContainer" containerID="2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d"
Sep 17 18:14:36.823779 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.823763    2577 scope.go:117] "RemoveContainer" containerID="79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e"
Sep 17 18:14:36.831186 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.831172    2577 scope.go:117] "RemoveContainer" containerID="bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d"
Sep 17 18:14:36.838676 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.838658    2577 scope.go:117] "RemoveContainer" containerID="082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187"
Sep 17 18:14:36.838975 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:14:36.838959    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187\": container with ID starting with 082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187 not found: ID does not exist" containerID="082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187"
Sep 17 18:14:36.839025 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.838983    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187"} err="failed to get container status \"082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187\": rpc error: code = NotFound desc = could not find container \"082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187\": container with ID starting with 082533ea4a5a89cf6f7bdfb31fc2ca5ace45422c2d6b36a729e4eaf06a850187 not found: ID does not exist"
Sep 17 18:14:36.839025 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.838999    2577 scope.go:117] "RemoveContainer" containerID="2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d"
Sep 17 18:14:36.839293 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:14:36.839274    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d\": container with ID starting with 2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d not found: ID does not exist" containerID="2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d"
Sep 17 18:14:36.839358 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.839302    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d"} err="failed to get container status \"2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d\": rpc error: code = NotFound desc = could not find container \"2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d\": container with ID starting with 2e6f3cb1658c6b09646c1ae62291822886b654d273731389a6fccdd486a3903d not found: ID does not exist"
Sep 17 18:14:36.839358 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.839318    2577 scope.go:117] "RemoveContainer" containerID="79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e"
Sep 17 18:14:36.839619 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:14:36.839602    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e\": container with ID starting with 79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e not found: ID does not exist" containerID="79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e"
Sep 17 18:14:36.839709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.839621    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e"} err="failed to get container status \"79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e\": rpc error: code = NotFound desc = could not find container \"79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e\": container with ID starting with 79969652d2debfab11da616d91cb936df024eca981908bfe1fc9e831356adb6e not found: ID does not exist"
Sep 17 18:14:36.839709 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.839633    2577 scope.go:117] "RemoveContainer" containerID="bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d"
Sep 17 18:14:36.840010 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:14:36.839989    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d\": container with ID starting with bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d not found: ID does not exist" containerID="bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d"
Sep 17 18:14:36.840062 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.840014    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d"} err="failed to get container status \"bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d\": rpc error: code = NotFound desc = could not find container \"bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d\": container with ID starting with bc4e925e0fe4d95dc783bd75820b29ceb6f98681068b74581304c0876d526f0d not found: ID does not exist"
Sep 17 18:14:36.840443 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.840425    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"]
Sep 17 18:14:36.849727 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:36.849708    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-fc652-predictor-6fff76ccbf-8k5zd"]
Sep 17 18:14:37.585024 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:14:37.584990    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" path="/var/lib/kubelet/pods/e8046de1-3a6d-458f-a741-7637c42dfa48/volumes"
Sep 17 18:15:34.721359 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721323    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"]
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721671    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721682    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721695    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kube-rbac-proxy"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721701    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kube-rbac-proxy"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721715    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721726    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721732    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721744    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="storage-initializer"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721749    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="storage-initializer"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721757    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kserve-container"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721762    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kserve-container"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721813    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="agent"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721823    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kserve-container"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721831    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kube-rbac-proxy"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721836    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8046de1-3a6d-458f-a741-7637c42dfa48" containerName="kube-rbac-proxy"
Sep 17 18:15:34.721871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.721843    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5332680-2e45-4515-8f92-ed7c03120327" containerName="kserve-container"
Sep 17 18:15:34.724794 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.724778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.732357 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.732341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-5bb89-predictor-serving-cert\""
Sep 17 18:15:34.732580 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.732546    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:15:34.732640 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.732551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-5bb89-kube-rbac-proxy-sar-config\""
Sep 17 18:15:34.732690 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.732649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:15:34.732690 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.732650    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:15:34.734109 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.734091    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:15:34.734257 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.734134    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:15:34.736163 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.736144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"]
Sep 17 18:15:34.777944 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.777909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.777944 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.777948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.778142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.777968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-cabundle-cert\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.778142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.778073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86j54\" (UniqueName: \"kubernetes.io/projected/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-api-access-86j54\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.778142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.778108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kserve-provision-location\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.878808 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.878768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.878808 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.878816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.879050 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.878838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-cabundle-cert\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.879050 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.878908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86j54\" (UniqueName: \"kubernetes.io/projected/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-api-access-86j54\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.879050 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.878933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kserve-provision-location\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.879050 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:15:34.878940    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-serving-cert: secret "raw-sklearn-runtime-5bb89-predictor-serving-cert" not found
Sep 17 18:15:34.879050 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:15:34.879021    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls podName:4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d nodeName:}" failed. No retries permitted until 2026-09-17 18:15:35.378999522 +0000 UTC m=+1122.222162661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls") pod "raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" (UID: "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d") : secret "raw-sklearn-runtime-5bb89-predictor-serving-cert" not found
Sep 17 18:15:34.879327 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.879306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kserve-provision-location\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.879502 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.879479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.879541 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.879496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-cabundle-cert\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:34.889401 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:34.889371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86j54\" (UniqueName: \"kubernetes.io/projected/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-api-access-86j54\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:35.383758 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:35.383722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:35.386197 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:35.386175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls\") pod \"raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:35.635205 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:35.635124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:35.907200 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:35.907047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"]
Sep 17 18:15:35.909821 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:15:35.909791    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdd3bc6_ac63_4c20_b85a_c0c9f43f5d3d.slice/crio-7c4e30ac5fddef1a1729afd4744c429b83cb9746784c0e3b756eda45acfe5db3 WatchSource:0}: Error finding container 7c4e30ac5fddef1a1729afd4744c429b83cb9746784c0e3b756eda45acfe5db3: Status 404 returned error can't find the container with id 7c4e30ac5fddef1a1729afd4744c429b83cb9746784c0e3b756eda45acfe5db3
Sep 17 18:15:36.080258 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:36.080228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerStarted","Data":"fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026"}
Sep 17 18:15:36.080435 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:36.080284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerStarted","Data":"7c4e30ac5fddef1a1729afd4744c429b83cb9746784c0e3b756eda45acfe5db3"}
Sep 17 18:15:40.099341 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:40.099307    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerID="fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026" exitCode=0
Sep 17 18:15:40.099836 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:40.099377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerDied","Data":"fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026"}
Sep 17 18:15:41.106332 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:41.106291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerStarted","Data":"d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a"}
Sep 17 18:15:41.106762 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:41.106383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerStarted","Data":"646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc"}
Sep 17 18:15:41.106762 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:41.106588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:41.127816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:41.127774    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podStartSLOduration=3.249974575 podStartE2EDuration="7.127759459s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877784884s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:41.125698914 +0000 UTC m=+1127.968862062" watchObservedRunningTime="2026-09-17 18:15:41.127759459 +0000 UTC m=+1127.970922607"
Sep 17 18:15:42.112112 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:42.112081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:42.113325 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:42.113298    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:15:43.116422 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:43.116383    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:15:48.121418 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:48.121384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:15:48.121922 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:48.121856    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:15:58.122847 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:15:58.122808    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:08.121869 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:08.121786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:18.121931 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:18.121881    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:28.122463 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:28.122418    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:38.122369 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:38.122318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:48.122411 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:48.122373    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:52.579672 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:52.579630    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:55.829231 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:55.829124    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:16:55.833278 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:16:55.831970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:17:01.855676 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.855642    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"]
Sep 17 18:17:01.858822 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.858806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:01.861329 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.861307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:17:01.861445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.861377    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:17:01.868154 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.868132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"]
Sep 17 18:17:01.920070 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.920035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwrs\" (UniqueName: \"kubernetes.io/projected/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-api-access-vkwrs\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:01.920070 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.920072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29f5f9f4-7677-45ea-8b56-edc12b09a717-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:01.920284 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.920169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f5f9f4-7677-45ea-8b56-edc12b09a717-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:01.920284 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.920188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:01.920284 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:01.920219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.021269 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.021237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.021460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.021276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.021460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.021318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwrs\" (UniqueName: \"kubernetes.io/projected/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-api-access-vkwrs\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.021460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.021337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29f5f9f4-7677-45ea-8b56-edc12b09a717-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.021460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.021375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f5f9f4-7677-45ea-8b56-edc12b09a717-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.021857 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.021835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29f5f9f4-7677-45ea-8b56-edc12b09a717-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.022136 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.022116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.022181 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.022116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.023871 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.023850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f5f9f4-7677-45ea-8b56-edc12b09a717-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.034529 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.034508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwrs\" (UniqueName: \"kubernetes.io/projected/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-api-access-vkwrs\") pod \"isvc-canary-create-predictor-5956b77f85-mvn7z\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.169475 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.169440    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:02.443300 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.443276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"]
Sep 17 18:17:02.446219 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:17:02.446195    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f5f9f4_7677_45ea_8b56_edc12b09a717.slice/crio-ebd4a3a74815002db073f57c076fa73f2a6ed658a6a34f4d1445d9eac1a8b12b WatchSource:0}: Error finding container ebd4a3a74815002db073f57c076fa73f2a6ed658a6a34f4d1445d9eac1a8b12b: Status 404 returned error can't find the container with id ebd4a3a74815002db073f57c076fa73f2a6ed658a6a34f4d1445d9eac1a8b12b
Sep 17 18:17:02.486085 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.486049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerStarted","Data":"ebd4a3a74815002db073f57c076fa73f2a6ed658a6a34f4d1445d9eac1a8b12b"}
Sep 17 18:17:02.580336 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:02.580304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:17:03.491764 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:03.491729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerStarted","Data":"09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92"}
Sep 17 18:17:04.776284 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:04.776247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"]
Sep 17 18:17:04.776742 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:04.776466    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container" containerID="cri-o://646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc" gracePeriod=30
Sep 17 18:17:04.776742 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:04.776518    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kube-rbac-proxy" containerID="cri-o://d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a" gracePeriod=30
Sep 17 18:17:05.503986 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:05.503947    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerID="d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a" exitCode=2
Sep 17 18:17:05.504162 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:05.504017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerDied","Data":"d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a"}
Sep 17 18:17:07.514888 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:07.514854    2577 generic.go:356] "Generic (PLEG): container finished" podID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerID="09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92" exitCode=0
Sep 17 18:17:07.515301 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:07.514900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerDied","Data":"09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92"}
Sep 17 18:17:08.117460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:08.117409    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 18:17:08.521781 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:08.521749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerStarted","Data":"24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39"}
Sep 17 18:17:08.522164 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:08.521806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerStarted","Data":"727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd"}
Sep 17 18:17:08.522164 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:08.522005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:08.542150 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:08.542105    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podStartSLOduration=3.466639049 podStartE2EDuration="7.54209079s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.075451741s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:08.540854789 +0000 UTC m=+1215.384017938" watchObservedRunningTime="2026-09-17 18:17:08.54209079 +0000 UTC m=+1215.385253938"
Sep 17 18:17:09.526279 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:09.526243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:09.527813 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:09.527784    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:17:09.925697 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:09.925667    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:17:10.092982 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.092885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls\") pod \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") "
Sep 17 18:17:10.092982 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.092927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kserve-provision-location\") pod \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") "
Sep 17 18:17:10.092982 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.092976    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-cabundle-cert\") pod \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") "
Sep 17 18:17:10.093274 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.092992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86j54\" (UniqueName: \"kubernetes.io/projected/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-api-access-86j54\") pod \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") "
Sep 17 18:17:10.093274 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.093009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-rbac-proxy-sar-config\") pod \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\" (UID: \"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d\") "
Sep 17 18:17:10.093389 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.093362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" (UID: "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:10.093445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.093421    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" (UID: "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:10.093495 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.093422    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" (UID: "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:10.095141 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.095118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-api-access-86j54" (OuterVolumeSpecName: "kube-api-access-86j54") pod "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" (UID: "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d"). InnerVolumeSpecName "kube-api-access-86j54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:10.095209 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.095139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" (UID: "4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:10.193712 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.193673    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:10.193712 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.193706    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86j54\" (UniqueName: \"kubernetes.io/projected/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-api-access-86j54\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:10.193712 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.193720    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:10.193949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.193731    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:10.193949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.193743    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:17:10.532479 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.532446    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerID="646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc" exitCode=0
Sep 17 18:17:10.532911 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.532504    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"
Sep 17 18:17:10.532911 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.532531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerDied","Data":"646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc"}
Sep 17 18:17:10.532911 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.532604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs" event={"ID":"4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d","Type":"ContainerDied","Data":"7c4e30ac5fddef1a1729afd4744c429b83cb9746784c0e3b756eda45acfe5db3"}
Sep 17 18:17:10.532911 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.532633    2577 scope.go:117] "RemoveContainer" containerID="d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a"
Sep 17 18:17:10.533239 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.533213    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:17:10.543865 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.543838    2577 scope.go:117] "RemoveContainer" containerID="646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc"
Sep 17 18:17:10.551628 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.551607    2577 scope.go:117] "RemoveContainer" containerID="fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026"
Sep 17 18:17:10.556203 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.556179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"]
Sep 17 18:17:10.558516 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.558496    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5bb89-predictor-6dddf4b9cb-s4zzs"]
Sep 17 18:17:10.560773 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.560752    2577 scope.go:117] "RemoveContainer" containerID="d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a"
Sep 17 18:17:10.561064 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:17:10.561047    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a\": container with ID starting with d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a not found: ID does not exist" containerID="d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a"
Sep 17 18:17:10.561127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.561074    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a"} err="failed to get container status \"d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a\": rpc error: code = NotFound desc = could not find container \"d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a\": container with ID starting with d347d1203136a327bf88da928054f723449180452f372da75592146e4d31772a not found: ID does not exist"
Sep 17 18:17:10.561127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.561091    2577 scope.go:117] "RemoveContainer" containerID="646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc"
Sep 17 18:17:10.561350 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:17:10.561335    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc\": container with ID starting with 646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc not found: ID does not exist" containerID="646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc"
Sep 17 18:17:10.561389 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.561352    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc"} err="failed to get container status \"646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc\": rpc error: code = NotFound desc = could not find container \"646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc\": container with ID starting with 646f47b7fab13c93397b5f4d78cc1fa78235b2b20bb60519d5e564042c4c54bc not found: ID does not exist"
Sep 17 18:17:10.561389 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.561365    2577 scope.go:117] "RemoveContainer" containerID="fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026"
Sep 17 18:17:10.561687 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:17:10.561668    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026\": container with ID starting with fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026 not found: ID does not exist" containerID="fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026"
Sep 17 18:17:10.561770 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:10.561688    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026"} err="failed to get container status \"fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026\": rpc error: code = NotFound desc = could not find container \"fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026\": container with ID starting with fa15d3076079e8ef08b9641fce7bae7592c1769823523d50b445ff2f05e13026 not found: ID does not exist"
Sep 17 18:17:11.584747 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:11.584710    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" path="/var/lib/kubelet/pods/4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d/volumes"
Sep 17 18:17:15.537884 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:15.537853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:17:15.538392 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:15.538363    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:17:25.538882 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:25.538844    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:17:35.539250 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:35.539215    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:17:45.539041 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:45.538953    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:17:55.538736 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:17:55.538688    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:05.538781 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:05.538742    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:15.538333 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:15.538293    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:25.539730 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:25.539700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:18:28.175689 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:28.175653    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"]
Sep 17 18:18:28.176163 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:28.175916    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container" containerID="cri-o://727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd" gracePeriod=30
Sep 17 18:18:28.176163 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:28.175956    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kube-rbac-proxy" containerID="cri-o://24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39" gracePeriod=30
Sep 17 18:18:28.897267 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:28.897235    2577 generic.go:356] "Generic (PLEG): container finished" podID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerID="24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39" exitCode=2
Sep 17 18:18:28.897429 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:28.897307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerDied","Data":"24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39"}
Sep 17 18:18:30.534007 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:30.533957    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 18:18:32.821352 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.821330    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:18:32.887253 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f5f9f4-7677-45ea-8b56-edc12b09a717-proxy-tls\") pod \"29f5f9f4-7677-45ea-8b56-edc12b09a717\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") "
Sep 17 18:18:32.887434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-rbac-proxy-sar-config\") pod \"29f5f9f4-7677-45ea-8b56-edc12b09a717\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") "
Sep 17 18:18:32.887434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29f5f9f4-7677-45ea-8b56-edc12b09a717-kserve-provision-location\") pod \"29f5f9f4-7677-45ea-8b56-edc12b09a717\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") "
Sep 17 18:18:32.887434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-cabundle-cert\") pod \"29f5f9f4-7677-45ea-8b56-edc12b09a717\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") "
Sep 17 18:18:32.887434 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkwrs\" (UniqueName: \"kubernetes.io/projected/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-api-access-vkwrs\") pod \"29f5f9f4-7677-45ea-8b56-edc12b09a717\" (UID: \"29f5f9f4-7677-45ea-8b56-edc12b09a717\") "
Sep 17 18:18:32.887690 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f5f9f4-7677-45ea-8b56-edc12b09a717-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29f5f9f4-7677-45ea-8b56-edc12b09a717" (UID: "29f5f9f4-7677-45ea-8b56-edc12b09a717"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:32.887690 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29f5f9f4-7677-45ea-8b56-edc12b09a717" (UID: "29f5f9f4-7677-45ea-8b56-edc12b09a717"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:32.887767 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.887731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29f5f9f4-7677-45ea-8b56-edc12b09a717" (UID: "29f5f9f4-7677-45ea-8b56-edc12b09a717"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:32.889415 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.889397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f5f9f4-7677-45ea-8b56-edc12b09a717-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29f5f9f4-7677-45ea-8b56-edc12b09a717" (UID: "29f5f9f4-7677-45ea-8b56-edc12b09a717"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:32.889472 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.889409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-api-access-vkwrs" (OuterVolumeSpecName: "kube-api-access-vkwrs") pod "29f5f9f4-7677-45ea-8b56-edc12b09a717" (UID: "29f5f9f4-7677-45ea-8b56-edc12b09a717"). InnerVolumeSpecName "kube-api-access-vkwrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:32.918267 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.918234    2577 generic.go:356] "Generic (PLEG): container finished" podID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerID="727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd" exitCode=0
Sep 17 18:18:32.918419 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.918290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"
Sep 17 18:18:32.918419 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.918317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerDied","Data":"727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd"}
Sep 17 18:18:32.918419 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.918383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z" event={"ID":"29f5f9f4-7677-45ea-8b56-edc12b09a717","Type":"ContainerDied","Data":"ebd4a3a74815002db073f57c076fa73f2a6ed658a6a34f4d1445d9eac1a8b12b"}
Sep 17 18:18:32.918419 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.918409    2577 scope.go:117] "RemoveContainer" containerID="24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39"
Sep 17 18:18:32.927365 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.927349    2577 scope.go:117] "RemoveContainer" containerID="727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd"
Sep 17 18:18:32.935343 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.935326    2577 scope.go:117] "RemoveContainer" containerID="09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92"
Sep 17 18:18:32.941524 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.941501    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"]
Sep 17 18:18:32.943217 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.943196    2577 scope.go:117] "RemoveContainer" containerID="24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39"
Sep 17 18:18:32.943503 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:18:32.943484    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39\": container with ID starting with 24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39 not found: ID does not exist" containerID="24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39"
Sep 17 18:18:32.943571 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.943512    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39"} err="failed to get container status \"24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39\": rpc error: code = NotFound desc = could not find container \"24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39\": container with ID starting with 24357c951894f8c6807093ca59733a3bc458e8d07f54980fc2ea755ee35dfa39 not found: ID does not exist"
Sep 17 18:18:32.943571 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.943530    2577 scope.go:117] "RemoveContainer" containerID="727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd"
Sep 17 18:18:32.943803 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:18:32.943781    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd\": container with ID starting with 727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd not found: ID does not exist" containerID="727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd"
Sep 17 18:18:32.943881 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.943814    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd"} err="failed to get container status \"727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd\": rpc error: code = NotFound desc = could not find container \"727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd\": container with ID starting with 727392ee38eefc66fec4006eca5d9a53369dae13f5d5dbd73114ced0b405ffcd not found: ID does not exist"
Sep 17 18:18:32.943881 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.943837    2577 scope.go:117] "RemoveContainer" containerID="09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92"
Sep 17 18:18:32.944089 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:18:32.944073    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92\": container with ID starting with 09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92 not found: ID does not exist" containerID="09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92"
Sep 17 18:18:32.944139 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.944095    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92"} err="failed to get container status \"09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92\": rpc error: code = NotFound desc = could not find container \"09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92\": container with ID starting with 09b2422d34d66a6dadfec1f118d0012f18c52ed8ca8d5ceb2805047ab40e9e92 not found: ID does not exist"
Sep 17 18:18:32.944766 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.944750    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-mvn7z"]
Sep 17 18:18:32.988596 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.988553    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:32.988596 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.988594    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkwrs\" (UniqueName: \"kubernetes.io/projected/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-api-access-vkwrs\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:32.988725 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.988603    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29f5f9f4-7677-45ea-8b56-edc12b09a717-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:32.988725 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.988612    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29f5f9f4-7677-45ea-8b56-edc12b09a717-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:32.988725 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:32.988621    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29f5f9f4-7677-45ea-8b56-edc12b09a717-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:18:33.585423 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:33.585390    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" path="/var/lib/kubelet/pods/29f5f9f4-7677-45ea-8b56-edc12b09a717/volumes"
Sep 17 18:18:37.088067 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088029    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"]
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088535    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088556    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088610    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kube-rbac-proxy"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088619    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kube-rbac-proxy"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088633    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="storage-initializer"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088643    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="storage-initializer"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088654    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088674    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kube-rbac-proxy"
Sep 17 18:18:37.088678 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088682    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kube-rbac-proxy"
Sep 17 18:18:37.089260 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088704    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="storage-initializer"
Sep 17 18:18:37.089260 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088713    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="storage-initializer"
Sep 17 18:18:37.089260 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088829    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kserve-container"
Sep 17 18:18:37.089260 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088842    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kube-rbac-proxy"
Sep 17 18:18:37.089260 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088853    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bdd3bc6-ac63-4c20-b85a-c0c9f43f5d3d" containerName="kube-rbac-proxy"
Sep 17 18:18:37.089260 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.088863    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f5f9f4-7677-45ea-8b56-edc12b09a717" containerName="kserve-container"
Sep 17 18:18:37.093851 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.093829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"]
Sep 17 18:18:37.093982 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.093946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.096961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.096937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:18:37.096961 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.096954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:37.097152 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.096946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:37.097152 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.097067    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:18:37.097289 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.097272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.097353 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.097324    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:18:37.098112 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.098089    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:18:37.098207 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.098146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:18:37.100948 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.100926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:18:37.102148 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.102129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"]
Sep 17 18:18:37.104794 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.104773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"]
Sep 17 18:18:37.124302 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.124271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.124418 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.124305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.124418 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.124343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f4536cd-45cb-405d-b09c-ae85b79dd560-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.124418 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.124389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nz9\" (UniqueName: \"kubernetes.io/projected/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-api-access-j2nz9\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.124525 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.124464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f4536cd-45cb-405d-b09c-ae85b79dd560-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225207 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f4536cd-45cb-405d-b09c-ae85b79dd560-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225207 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nz9\" (UniqueName: \"kubernetes.io/projected/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-api-access-j2nz9\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2df4540-d09d-45b4-aa30-a303757392d8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.225473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.225473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65qn\" (UniqueName: \"kubernetes.io/projected/b2df4540-d09d-45b4-aa30-a303757392d8-kube-api-access-d65qn\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.225473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.225473 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f4536cd-45cb-405d-b09c-ae85b79dd560-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f4536cd-45cb-405d-b09c-ae85b79dd560-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.225782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.225644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.226138 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.226117    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.226207 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.226150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.227989 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.227966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f4536cd-45cb-405d-b09c-ae85b79dd560-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.233542 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.233518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nz9\" (UniqueName: \"kubernetes.io/projected/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-api-access-j2nz9\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.326933 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.326894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.326953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2df4540-d09d-45b4-aa30-a303757392d8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.326971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.326995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65qn\" (UniqueName: \"kubernetes.io/projected/b2df4540-d09d-45b4-aa30-a303757392d8-kube-api-access-d65qn\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.327013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327130 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:18:37.327040    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:18:37.327130 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:18:37.327129    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls podName:b2df4540-d09d-45b4-aa30-a303757392d8 nodeName:}" failed. No retries permitted until 2026-09-17 18:18:37.827109832 +0000 UTC m=+1304.670272959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls") pod "isvc-canary-promote-predictor-7877c757df-4mrdt" (UID: "b2df4540-d09d-45b4-aa30-a303757392d8") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:18:37.327460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.327371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2df4540-d09d-45b4-aa30-a303757392d8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327763 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.327739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.327763 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.327752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.336069 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.336049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65qn\" (UniqueName: \"kubernetes.io/projected/b2df4540-d09d-45b4-aa30-a303757392d8-kube-api-access-d65qn\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.407895 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.407864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:37.681370 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.681342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"]
Sep 17 18:18:37.683188 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:18:37.683164    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4536cd_45cb_405d_b09c_ae85b79dd560.slice/crio-58f5ac0bfef7ba45eb83cbd2ebf4f2349e52aed45824cfd15985c990ea7ba7d9 WatchSource:0}: Error finding container 58f5ac0bfef7ba45eb83cbd2ebf4f2349e52aed45824cfd15985c990ea7ba7d9: Status 404 returned error can't find the container with id 58f5ac0bfef7ba45eb83cbd2ebf4f2349e52aed45824cfd15985c990ea7ba7d9
Sep 17 18:18:37.830440 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.830405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.832748 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.832719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-4mrdt\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:37.941962 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.941874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerStarted","Data":"cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0"}
Sep 17 18:18:37.941962 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:37.941939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerStarted","Data":"58f5ac0bfef7ba45eb83cbd2ebf4f2349e52aed45824cfd15985c990ea7ba7d9"}
Sep 17 18:18:38.013952 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:38.013919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:38.292138 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:38.292111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"]
Sep 17 18:18:38.294319 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:18:38.294293    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2df4540_d09d_45b4_aa30_a303757392d8.slice/crio-0b81c89dad6a3223fbdbbcaa1e30e7062dde955e854c443af6d5b30c6a788484 WatchSource:0}: Error finding container 0b81c89dad6a3223fbdbbcaa1e30e7062dde955e854c443af6d5b30c6a788484: Status 404 returned error can't find the container with id 0b81c89dad6a3223fbdbbcaa1e30e7062dde955e854c443af6d5b30c6a788484
Sep 17 18:18:38.947590 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:38.947535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerStarted","Data":"a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb"}
Sep 17 18:18:38.947796 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:38.947627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerStarted","Data":"0b81c89dad6a3223fbdbbcaa1e30e7062dde955e854c443af6d5b30c6a788484"}
Sep 17 18:18:41.963089 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:41.963053    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerID="cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0" exitCode=0
Sep 17 18:18:41.963455 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:41.963116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerDied","Data":"cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0"}
Sep 17 18:18:42.969136 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.969104    2577 generic.go:356] "Generic (PLEG): container finished" podID="b2df4540-d09d-45b4-aa30-a303757392d8" containerID="a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb" exitCode=0
Sep 17 18:18:42.969590 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.969180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerDied","Data":"a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb"}
Sep 17 18:18:42.971614 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.971589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerStarted","Data":"735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725"}
Sep 17 18:18:42.971769 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.971645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerStarted","Data":"707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880"}
Sep 17 18:18:42.971920 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.971894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:42.972029 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.971961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:42.973598 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:42.973556    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:43.009503 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.009461    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podStartSLOduration=2.386335499 podStartE2EDuration="6.009446798s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.623111299s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:43.006991051 +0000 UTC m=+1309.850154198" watchObservedRunningTime="2026-09-17 18:18:43.009446798 +0000 UTC m=+1309.852609967"
Sep 17 18:18:43.978971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.978939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerStarted","Data":"0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b"}
Sep 17 18:18:43.979375 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.979001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerStarted","Data":"6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc"}
Sep 17 18:18:43.979375 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.979239    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:43.979476 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.979454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:43.979619 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.979602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:43.980668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.980648    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:18:43.999532 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:43.999492    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podStartSLOduration=3.158010854 podStartE2EDuration="6.999479832s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841468978s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:43.998094281 +0000 UTC m=+1310.841257430" watchObservedRunningTime="2026-09-17 18:18:43.999479832 +0000 UTC m=+1310.842642980"
Sep 17 18:18:44.983538 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:44.983499    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:18:48.984149 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:48.984123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:18:48.984601 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:48.984557    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:49.988053 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:49.988025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:18:49.988744 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:49.988714    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:18:58.984730 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:58.984689    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:59.988785 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:18:59.988745    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:08.984904 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:08.984825    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:19:09.988800 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:09.988765    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:18.985452 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:18.985412    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:19:19.988589 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:19.988520    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:28.984720 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:28.984682    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:19:29.988986 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:29.988950    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:38.985410 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:38.985371    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:19:39.988616 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:39.988555    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:48.985254 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:48.985217    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:19:49.989635 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:49.989593    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:58.985184 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:58.985152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:19:59.103200 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.103169    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"]
Sep 17 18:19:59.103441 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.103411    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container" containerID="cri-o://6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc" gracePeriod=30
Sep 17 18:19:59.103605 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.103506    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kube-rbac-proxy" containerID="cri-o://0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b" gracePeriod=30
Sep 17 18:19:59.104415 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.104391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:19:59.351419 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.351329    2577 generic.go:356] "Generic (PLEG): container finished" podID="b2df4540-d09d-45b4-aa30-a303757392d8" containerID="0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b" exitCode=2
Sep 17 18:19:59.351419 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.351407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerDied","Data":"0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b"}
Sep 17 18:19:59.984460 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:19:59.984418    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:20:03.851580 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.851543    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:20:03.958526 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-cabundle-cert\") pod \"b2df4540-d09d-45b4-aa30-a303757392d8\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") "
Sep 17 18:20:03.958526 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d65qn\" (UniqueName: \"kubernetes.io/projected/b2df4540-d09d-45b4-aa30-a303757392d8-kube-api-access-d65qn\") pod \"b2df4540-d09d-45b4-aa30-a303757392d8\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") "
Sep 17 18:20:03.958526 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls\") pod \"b2df4540-d09d-45b4-aa30-a303757392d8\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") "
Sep 17 18:20:03.958852 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2df4540-d09d-45b4-aa30-a303757392d8-kserve-provision-location\") pod \"b2df4540-d09d-45b4-aa30-a303757392d8\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") "
Sep 17 18:20:03.958852 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958620    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-kube-rbac-proxy-sar-config\") pod \"b2df4540-d09d-45b4-aa30-a303757392d8\" (UID: \"b2df4540-d09d-45b4-aa30-a303757392d8\") "
Sep 17 18:20:03.959013 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2df4540-d09d-45b4-aa30-a303757392d8" (UID: "b2df4540-d09d-45b4-aa30-a303757392d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:03.959013 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.958999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2df4540-d09d-45b4-aa30-a303757392d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2df4540-d09d-45b4-aa30-a303757392d8" (UID: "b2df4540-d09d-45b4-aa30-a303757392d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:03.959170 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.959053    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2df4540-d09d-45b4-aa30-a303757392d8" (UID: "b2df4540-d09d-45b4-aa30-a303757392d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:03.960769 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.960749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2df4540-d09d-45b4-aa30-a303757392d8" (UID: "b2df4540-d09d-45b4-aa30-a303757392d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:03.960829 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:03.960770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2df4540-d09d-45b4-aa30-a303757392d8-kube-api-access-d65qn" (OuterVolumeSpecName: "kube-api-access-d65qn") pod "b2df4540-d09d-45b4-aa30-a303757392d8" (UID: "b2df4540-d09d-45b4-aa30-a303757392d8"). InnerVolumeSpecName "kube-api-access-d65qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:04.059696 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.059668    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d65qn\" (UniqueName: \"kubernetes.io/projected/b2df4540-d09d-45b4-aa30-a303757392d8-kube-api-access-d65qn\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:04.059696 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.059693    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2df4540-d09d-45b4-aa30-a303757392d8-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:04.059696 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.059703    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2df4540-d09d-45b4-aa30-a303757392d8-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:04.059696 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.059712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:04.059958 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.059721    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2df4540-d09d-45b4-aa30-a303757392d8-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:20:04.383444 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.383357    2577 generic.go:356] "Generic (PLEG): container finished" podID="b2df4540-d09d-45b4-aa30-a303757392d8" containerID="6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc" exitCode=0
Sep 17 18:20:04.383444 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.383405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"
Sep 17 18:20:04.383675 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.383438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerDied","Data":"6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc"}
Sep 17 18:20:04.383675 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.383498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt" event={"ID":"b2df4540-d09d-45b4-aa30-a303757392d8","Type":"ContainerDied","Data":"0b81c89dad6a3223fbdbbcaa1e30e7062dde955e854c443af6d5b30c6a788484"}
Sep 17 18:20:04.383675 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.383526    2577 scope.go:117] "RemoveContainer" containerID="0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b"
Sep 17 18:20:04.392513 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.392496    2577 scope.go:117] "RemoveContainer" containerID="6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc"
Sep 17 18:20:04.400252 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.400235    2577 scope.go:117] "RemoveContainer" containerID="a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb"
Sep 17 18:20:04.406877 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.406854    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"]
Sep 17 18:20:04.407963 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.407945    2577 scope.go:117] "RemoveContainer" containerID="0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b"
Sep 17 18:20:04.408254 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:20:04.408234    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b\": container with ID starting with 0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b not found: ID does not exist" containerID="0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b"
Sep 17 18:20:04.408339 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.408266    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b"} err="failed to get container status \"0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b\": rpc error: code = NotFound desc = could not find container \"0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b\": container with ID starting with 0cbfc29e76e2232e0e12f74f3060c3cab0229a83971001be6f6541dde094b32b not found: ID does not exist"
Sep 17 18:20:04.408339 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.408291    2577 scope.go:117] "RemoveContainer" containerID="6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc"
Sep 17 18:20:04.408552 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:20:04.408535    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc\": container with ID starting with 6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc not found: ID does not exist" containerID="6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc"
Sep 17 18:20:04.408640 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.408582    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc"} err="failed to get container status \"6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc\": rpc error: code = NotFound desc = could not find container \"6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc\": container with ID starting with 6246978ea59454fda1ae579ea93060d70679e27248af237d41f3af10e18f26bc not found: ID does not exist"
Sep 17 18:20:04.408640 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.408604    2577 scope.go:117] "RemoveContainer" containerID="a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb"
Sep 17 18:20:04.408848 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:20:04.408833    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb\": container with ID starting with a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb not found: ID does not exist" containerID="a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb"
Sep 17 18:20:04.408898 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.408855    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb"} err="failed to get container status \"a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb\": rpc error: code = NotFound desc = could not find container \"a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb\": container with ID starting with a3770ff643a201cfd4f1ba88ed4888c16f0aba1c4f63f195242de9d328125cbb not found: ID does not exist"
Sep 17 18:20:04.409209 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:04.409191    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-4mrdt"]
Sep 17 18:20:05.586154 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:20:05.586123    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" path="/var/lib/kubelet/pods/b2df4540-d09d-45b4-aa30-a303757392d8/volumes"
Sep 17 18:21:37.434585 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:37.434537    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"]
Sep 17 18:21:37.435127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:37.434784    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" containerID="cri-o://707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880" gracePeriod=30
Sep 17 18:21:37.435127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:37.434836    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kube-rbac-proxy" containerID="cri-o://735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725" gracePeriod=30
Sep 17 18:21:37.821161 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:37.821066    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerID="735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725" exitCode=2
Sep 17 18:21:37.821161 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:37.821136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerDied","Data":"735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725"}
Sep 17 18:21:38.979676 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:38.979635    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 18:21:38.985099 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:38.985073    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:21:42.080391 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.080361    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:21:42.150231 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150204    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f4536cd-45cb-405d-b09c-ae85b79dd560-kserve-provision-location\") pod \"2f4536cd-45cb-405d-b09c-ae85b79dd560\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") "
Sep 17 18:21:42.150365 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-cabundle-cert\") pod \"2f4536cd-45cb-405d-b09c-ae85b79dd560\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") "
Sep 17 18:21:42.150365 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f4536cd-45cb-405d-b09c-ae85b79dd560-proxy-tls\") pod \"2f4536cd-45cb-405d-b09c-ae85b79dd560\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") "
Sep 17 18:21:42.150365 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2nz9\" (UniqueName: \"kubernetes.io/projected/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-api-access-j2nz9\") pod \"2f4536cd-45cb-405d-b09c-ae85b79dd560\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") "
Sep 17 18:21:42.150488 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-rbac-proxy-sar-config\") pod \"2f4536cd-45cb-405d-b09c-ae85b79dd560\" (UID: \"2f4536cd-45cb-405d-b09c-ae85b79dd560\") "
Sep 17 18:21:42.150613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150584    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4536cd-45cb-405d-b09c-ae85b79dd560-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f4536cd-45cb-405d-b09c-ae85b79dd560" (UID: "2f4536cd-45cb-405d-b09c-ae85b79dd560"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:42.150736 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150710    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f4536cd-45cb-405d-b09c-ae85b79dd560-kserve-provision-location\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:42.150808 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f4536cd-45cb-405d-b09c-ae85b79dd560" (UID: "2f4536cd-45cb-405d-b09c-ae85b79dd560"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:42.150808 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.150792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f4536cd-45cb-405d-b09c-ae85b79dd560" (UID: "2f4536cd-45cb-405d-b09c-ae85b79dd560"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:42.152407 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.152381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4536cd-45cb-405d-b09c-ae85b79dd560-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f4536cd-45cb-405d-b09c-ae85b79dd560" (UID: "2f4536cd-45cb-405d-b09c-ae85b79dd560"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:42.152483 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.152430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-api-access-j2nz9" (OuterVolumeSpecName: "kube-api-access-j2nz9") pod "2f4536cd-45cb-405d-b09c-ae85b79dd560" (UID: "2f4536cd-45cb-405d-b09c-ae85b79dd560"). InnerVolumeSpecName "kube-api-access-j2nz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:42.252043 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.252012    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-cabundle-cert\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:42.252043 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.252040    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f4536cd-45cb-405d-b09c-ae85b79dd560-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:42.252043 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.252048    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nz9\" (UniqueName: \"kubernetes.io/projected/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-api-access-j2nz9\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:42.252262 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.252057    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f4536cd-45cb-405d-b09c-ae85b79dd560-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:21:42.852874 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.852843    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerID="707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880" exitCode=0
Sep 17 18:21:42.853077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.852904    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"
Sep 17 18:21:42.853077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.852919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerDied","Data":"707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880"}
Sep 17 18:21:42.853077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.852980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx" event={"ID":"2f4536cd-45cb-405d-b09c-ae85b79dd560","Type":"ContainerDied","Data":"58f5ac0bfef7ba45eb83cbd2ebf4f2349e52aed45824cfd15985c990ea7ba7d9"}
Sep 17 18:21:42.853077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.853008    2577 scope.go:117] "RemoveContainer" containerID="735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725"
Sep 17 18:21:42.862061 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.862042    2577 scope.go:117] "RemoveContainer" containerID="707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880"
Sep 17 18:21:42.870734 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.870708    2577 scope.go:117] "RemoveContainer" containerID="cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0"
Sep 17 18:21:42.877308 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.877286    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"]
Sep 17 18:21:42.878634 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.878620    2577 scope.go:117] "RemoveContainer" containerID="735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725"
Sep 17 18:21:42.878932 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:21:42.878914    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725\": container with ID starting with 735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725 not found: ID does not exist" containerID="735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725"
Sep 17 18:21:42.878983 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.878945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725"} err="failed to get container status \"735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725\": rpc error: code = NotFound desc = could not find container \"735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725\": container with ID starting with 735bdd005005490558457b2b8f8d73f56a1b708981524e991fd6a758d6cf8725 not found: ID does not exist"
Sep 17 18:21:42.878983 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.878968    2577 scope.go:117] "RemoveContainer" containerID="707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880"
Sep 17 18:21:42.879181 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:21:42.879165    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880\": container with ID starting with 707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880 not found: ID does not exist" containerID="707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880"
Sep 17 18:21:42.879220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.879184    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880"} err="failed to get container status \"707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880\": rpc error: code = NotFound desc = could not find container \"707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880\": container with ID starting with 707117ca9c6fc0bc5a9bc6b91e0f0d66736ce3416fdadcfc75d7f34ee5295880 not found: ID does not exist"
Sep 17 18:21:42.879220 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.879198    2577 scope.go:117] "RemoveContainer" containerID="cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0"
Sep 17 18:21:42.879400 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:21:42.879383    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0\": container with ID starting with cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0 not found: ID does not exist" containerID="cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0"
Sep 17 18:21:42.879445 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.879406    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0"} err="failed to get container status \"cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0\": rpc error: code = NotFound desc = could not find container \"cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0\": container with ID starting with cd09ef7344b3a4815793da4dc558eb1dcc1590cf008b0cbc31419348b4ce6ca0 not found: ID does not exist"
Sep 17 18:21:42.882706 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:42.882684    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-xkrvx"]
Sep 17 18:21:43.584636 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:43.584605    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" path="/var/lib/kubelet/pods/2f4536cd-45cb-405d-b09c-ae85b79dd560/volumes"
Sep 17 18:21:51.822185 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822139    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"]
Sep 17 18:21:51.822698 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822678    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="storage-initializer"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822699    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="storage-initializer"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822710    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822726    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822738    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="storage-initializer"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822746    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="storage-initializer"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822757    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822765    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822775    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kube-rbac-proxy"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822784    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kube-rbac-proxy"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822798    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kube-rbac-proxy"
Sep 17 18:21:51.822804 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822805    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kube-rbac-proxy"
Sep 17 18:21:51.823373 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822926    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kserve-container"
Sep 17 18:21:51.823373 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822939    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kserve-container"
Sep 17 18:21:51.823373 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822953    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4536cd-45cb-405d-b09c-ae85b79dd560" containerName="kube-rbac-proxy"
Sep 17 18:21:51.823373 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.822961    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2df4540-d09d-45b4-aa30-a303757392d8" containerName="kube-rbac-proxy"
Sep 17 18:21:51.826028 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.826009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:51.828847 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.828827    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:21:51.829032 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.828865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 18:21:51.829233 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.829212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 18:21:51.829317 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.829253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:21:51.829375 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.829310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6q8k\""
Sep 17 18:21:51.834917 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.834889    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"]
Sep 17 18:21:51.930345 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.930313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz777\" (UniqueName: \"kubernetes.io/projected/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-api-access-pz777\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:51.930500 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.930368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:51.930500 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:51.930443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c60002b5-3e77-4688-bbc1-3367779d5fcc-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.031405 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.031370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.031616 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.031414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c60002b5-3e77-4688-bbc1-3367779d5fcc-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.031616 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.031488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz777\" (UniqueName: \"kubernetes.io/projected/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-api-access-pz777\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.032144 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.032125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.033796 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.033776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c60002b5-3e77-4688-bbc1-3367779d5fcc-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.039784 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.039758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz777\" (UniqueName: \"kubernetes.io/projected/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-api-access-pz777\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-lwcfv\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.136578 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.136477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.407767 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.407740    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"]
Sep 17 18:21:52.409961 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:21:52.409929    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60002b5_3e77_4688_bbc1_3367779d5fcc.slice/crio-df6ef49f48de5a6ed90c1a9d79b5ed2a6a1a45a64a244bb66cc5c859a463a955 WatchSource:0}: Error finding container df6ef49f48de5a6ed90c1a9d79b5ed2a6a1a45a64a244bb66cc5c859a463a955: Status 404 returned error can't find the container with id df6ef49f48de5a6ed90c1a9d79b5ed2a6a1a45a64a244bb66cc5c859a463a955
Sep 17 18:21:52.903183 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.903150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" event={"ID":"c60002b5-3e77-4688-bbc1-3367779d5fcc","Type":"ContainerStarted","Data":"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"}
Sep 17 18:21:52.903549 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.903206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" event={"ID":"c60002b5-3e77-4688-bbc1-3367779d5fcc","Type":"ContainerStarted","Data":"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"}
Sep 17 18:21:52.903549 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.903228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" event={"ID":"c60002b5-3e77-4688-bbc1-3367779d5fcc","Type":"ContainerStarted","Data":"df6ef49f48de5a6ed90c1a9d79b5ed2a6a1a45a64a244bb66cc5c859a463a955"}
Sep 17 18:21:52.903549 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.903251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:52.922140 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:52.922094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" podStartSLOduration=1.922081438 podStartE2EDuration="1.922081438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:52.920521454 +0000 UTC m=+1499.763684603" watchObservedRunningTime="2026-09-17 18:21:52.922081438 +0000 UTC m=+1499.765244585"
Sep 17 18:21:53.907577 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:53.907523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:53.909317 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:53.909297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:21:55.857551 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:55.857453    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:21:55.882016 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:21:55.869284    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:22:00.921137 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:22:00.921111    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:23:36.882191 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:36.882119    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-lwcfv_c60002b5-3e77-4688-bbc1-3367779d5fcc/kserve-container/0.log"
Sep 17 18:23:37.087811 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.087781    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"]
Sep 17 18:23:37.088018 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.087996    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kserve-container" containerID="cri-o://471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3" gracePeriod=30
Sep 17 18:23:37.088130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.088084    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kube-rbac-proxy" containerID="cri-o://5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3" gracePeriod=30
Sep 17 18:23:37.326476 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.326451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:23:37.367953 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.367927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz777\" (UniqueName: \"kubernetes.io/projected/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-api-access-pz777\") pod \"c60002b5-3e77-4688-bbc1-3367779d5fcc\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") "
Sep 17 18:23:37.368081 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.367963    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-rbac-proxy-sar-config\") pod \"c60002b5-3e77-4688-bbc1-3367779d5fcc\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") "
Sep 17 18:23:37.368081 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.367988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c60002b5-3e77-4688-bbc1-3367779d5fcc-proxy-tls\") pod \"c60002b5-3e77-4688-bbc1-3367779d5fcc\" (UID: \"c60002b5-3e77-4688-bbc1-3367779d5fcc\") "
Sep 17 18:23:37.368315 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.368294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c60002b5-3e77-4688-bbc1-3367779d5fcc" (UID: "c60002b5-3e77-4688-bbc1-3367779d5fcc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:37.369989 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.369958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-api-access-pz777" (OuterVolumeSpecName: "kube-api-access-pz777") pod "c60002b5-3e77-4688-bbc1-3367779d5fcc" (UID: "c60002b5-3e77-4688-bbc1-3367779d5fcc"). InnerVolumeSpecName "kube-api-access-pz777". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:37.370082 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.369990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60002b5-3e77-4688-bbc1-3367779d5fcc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c60002b5-3e77-4688-bbc1-3367779d5fcc" (UID: "c60002b5-3e77-4688-bbc1-3367779d5fcc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:37.382597 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382572    2577 generic.go:356] "Generic (PLEG): container finished" podID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerID="5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3" exitCode=2
Sep 17 18:23:37.382699 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382599    2577 generic.go:356] "Generic (PLEG): container finished" podID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerID="471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3" exitCode=2
Sep 17 18:23:37.382699 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382618    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"
Sep 17 18:23:37.382699 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" event={"ID":"c60002b5-3e77-4688-bbc1-3367779d5fcc","Type":"ContainerDied","Data":"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"}
Sep 17 18:23:37.382828 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" event={"ID":"c60002b5-3e77-4688-bbc1-3367779d5fcc","Type":"ContainerDied","Data":"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"}
Sep 17 18:23:37.382828 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv" event={"ID":"c60002b5-3e77-4688-bbc1-3367779d5fcc","Type":"ContainerDied","Data":"df6ef49f48de5a6ed90c1a9d79b5ed2a6a1a45a64a244bb66cc5c859a463a955"}
Sep 17 18:23:37.382828 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.382763    2577 scope.go:117] "RemoveContainer" containerID="5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"
Sep 17 18:23:37.391100 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.391085    2577 scope.go:117] "RemoveContainer" containerID="471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"
Sep 17 18:23:37.401076 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.401055    2577 scope.go:117] "RemoveContainer" containerID="5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"
Sep 17 18:23:37.401453 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:23:37.401425    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3\": container with ID starting with 5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3 not found: ID does not exist" containerID="5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"
Sep 17 18:23:37.401604 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.401464    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"} err="failed to get container status \"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3\": rpc error: code = NotFound desc = could not find container \"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3\": container with ID starting with 5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3 not found: ID does not exist"
Sep 17 18:23:37.401604 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.401489    2577 scope.go:117] "RemoveContainer" containerID="471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"
Sep 17 18:23:37.401842 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:23:37.401820    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3\": container with ID starting with 471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3 not found: ID does not exist" containerID="471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"
Sep 17 18:23:37.401911 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.401852    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"} err="failed to get container status \"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3\": rpc error: code = NotFound desc = could not find container \"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3\": container with ID starting with 471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3 not found: ID does not exist"
Sep 17 18:23:37.401911 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.401874    2577 scope.go:117] "RemoveContainer" containerID="5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"
Sep 17 18:23:37.402161 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.402136    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3"} err="failed to get container status \"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3\": rpc error: code = NotFound desc = could not find container \"5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3\": container with ID starting with 5e7668b1c7eb526c9ae5e4736e96bc3fa3503daabcc2d30d9c5570744521c9b3 not found: ID does not exist"
Sep 17 18:23:37.402207 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.402162    2577 scope.go:117] "RemoveContainer" containerID="471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"
Sep 17 18:23:37.402410 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.402387    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3"} err="failed to get container status \"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3\": rpc error: code = NotFound desc = could not find container \"471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3\": container with ID starting with 471fa720bc4d07ae6d8d6a0be87f85714119ded5830ec882da126c6a2035c4b3 not found: ID does not exist"
Sep 17 18:23:37.403693 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.403674    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"]
Sep 17 18:23:37.408381 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.408362    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-lwcfv"]
Sep 17 18:23:37.469125 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.469066    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz777\" (UniqueName: \"kubernetes.io/projected/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-api-access-pz777\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:23:37.469125 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.469086    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c60002b5-3e77-4688-bbc1-3367779d5fcc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:23:37.469125 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.469096    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c60002b5-3e77-4688-bbc1-3367779d5fcc-proxy-tls\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:23:37.584839 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:37.584810    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" path="/var/lib/kubelet/pods/c60002b5-3e77-4688-bbc1-3367779d5fcc/volumes"
Sep 17 18:23:38.274266 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274236    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t9cqr/must-gather-bxzmf"]
Sep 17 18:23:38.274668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274604    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kserve-container"
Sep 17 18:23:38.274668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kserve-container"
Sep 17 18:23:38.274668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274626    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kube-rbac-proxy"
Sep 17 18:23:38.274668 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274631    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kube-rbac-proxy"
Sep 17 18:23:38.274835 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274687    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kube-rbac-proxy"
Sep 17 18:23:38.274835 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.274700    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60002b5-3e77-4688-bbc1-3367779d5fcc" containerName="kserve-container"
Sep 17 18:23:38.279130 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.279113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.281844 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.281823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t9cqr\"/\"kube-root-ca.crt\""
Sep 17 18:23:38.281998 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.281979    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t9cqr\"/\"openshift-service-ca.crt\""
Sep 17 18:23:38.282098 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.282035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t9cqr\"/\"default-dockercfg-j525l\""
Sep 17 18:23:38.287208 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.287183    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t9cqr/must-gather-bxzmf"]
Sep 17 18:23:38.375101 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.375068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85f6eadb-b7f4-4007-9b2c-792ec54744d1-must-gather-output\") pod \"must-gather-bxzmf\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") " pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.375272 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.375136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdmd\" (UniqueName: \"kubernetes.io/projected/85f6eadb-b7f4-4007-9b2c-792ec54744d1-kube-api-access-lmdmd\") pod \"must-gather-bxzmf\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") " pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.476100 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.476072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdmd\" (UniqueName: \"kubernetes.io/projected/85f6eadb-b7f4-4007-9b2c-792ec54744d1-kube-api-access-lmdmd\") pod \"must-gather-bxzmf\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") " pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.476261 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.476131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85f6eadb-b7f4-4007-9b2c-792ec54744d1-must-gather-output\") pod \"must-gather-bxzmf\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") " pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.476391 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.476378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85f6eadb-b7f4-4007-9b2c-792ec54744d1-must-gather-output\") pod \"must-gather-bxzmf\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") " pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.484156 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.484124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdmd\" (UniqueName: \"kubernetes.io/projected/85f6eadb-b7f4-4007-9b2c-792ec54744d1-kube-api-access-lmdmd\") pod \"must-gather-bxzmf\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") " pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.607365 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.607296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:23:38.883662 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.883633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t9cqr/must-gather-bxzmf"]
Sep 17 18:23:38.886347 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:23:38.886320    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85f6eadb_b7f4_4007_9b2c_792ec54744d1.slice/crio-7f626df5ab6585877275401434bc83f3f34eef20cb4e6168823382d7ed57b36f WatchSource:0}: Error finding container 7f626df5ab6585877275401434bc83f3f34eef20cb4e6168823382d7ed57b36f: Status 404 returned error can't find the container with id 7f626df5ab6585877275401434bc83f3f34eef20cb4e6168823382d7ed57b36f
Sep 17 18:23:38.888614 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:38.888597    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:23:39.393127 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:39.393088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9cqr/must-gather-bxzmf" event={"ID":"85f6eadb-b7f4-4007-9b2c-792ec54744d1","Type":"ContainerStarted","Data":"7f626df5ab6585877275401434bc83f3f34eef20cb4e6168823382d7ed57b36f"}
Sep 17 18:23:45.430715 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:45.430679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9cqr/must-gather-bxzmf" event={"ID":"85f6eadb-b7f4-4007-9b2c-792ec54744d1","Type":"ContainerStarted","Data":"ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f"}
Sep 17 18:23:45.431121 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:45.430733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9cqr/must-gather-bxzmf" event={"ID":"85f6eadb-b7f4-4007-9b2c-792ec54744d1","Type":"ContainerStarted","Data":"777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"}
Sep 17 18:23:45.446774 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:23:45.446710    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t9cqr/must-gather-bxzmf" podStartSLOduration=1.203856283 podStartE2EDuration="7.446688584s" totalImagesPullingTime="6.242832301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:45.445857753 +0000 UTC m=+1612.289020901" watchObservedRunningTime="2026-09-17 18:23:45.446688584 +0000 UTC m=+1612.289851733"
Sep 17 18:24:07.546374 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:07.546335    2577 generic.go:356] "Generic (PLEG): container finished" podID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerID="777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724" exitCode=0
Sep 17 18:24:07.546827 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:07.546409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9cqr/must-gather-bxzmf" event={"ID":"85f6eadb-b7f4-4007-9b2c-792ec54744d1","Type":"ContainerDied","Data":"777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"}
Sep 17 18:24:07.546827 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:07.546747    2577 scope.go:117] "RemoveContainer" containerID="777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"
Sep 17 18:24:07.629249 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:07.629218    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-t9cqr_must-gather-bxzmf_85f6eadb-b7f4-4007-9b2c-792ec54744d1/gather/0.log"
Sep 17 18:24:11.525077 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:11.525039    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j4qs8_4acb9960-a709-42b3-9f79-ef953d15e558/global-pull-secret-syncer/0.log"
Sep 17 18:24:11.630639 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:11.630609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bgkwf_5077d071-443f-4f95-b556-2ac43e51d310/konnectivity-agent/0.log"
Sep 17 18:24:11.736840 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:11.736812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-209.ec2.internal_799c76a8366626239baf308b23899d1e/haproxy/0.log"
Sep 17 18:24:13.109718 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.109684    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-t9cqr/must-gather-bxzmf"]
Sep 17 18:24:13.110179 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.109854    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-t9cqr/must-gather-bxzmf" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="copy" containerID="cri-o://ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f" gracePeriod=2
Sep 17 18:24:13.112253 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:24:13.112208    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxzmf\" is forbidden: User \"system:node:ip-10-0-143-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-t9cqr\": no relationship found between node 'ip-10-0-143-209.ec2.internal' and this object" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:24:13.113926 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.113357    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-t9cqr/must-gather-bxzmf"]
Sep 17 18:24:13.335510 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.335490    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-t9cqr_must-gather-bxzmf_85f6eadb-b7f4-4007-9b2c-792ec54744d1/copy/0.log"
Sep 17 18:24:13.337041 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.337027    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:24:13.339295 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:24:13.339274    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxzmf\" is forbidden: User \"system:node:ip-10-0-143-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-t9cqr\": no relationship found between node 'ip-10-0-143-209.ec2.internal' and this object" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:24:13.468203 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.468178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmdmd\" (UniqueName: \"kubernetes.io/projected/85f6eadb-b7f4-4007-9b2c-792ec54744d1-kube-api-access-lmdmd\") pod \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") "
Sep 17 18:24:13.468371 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.468231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85f6eadb-b7f4-4007-9b2c-792ec54744d1-must-gather-output\") pod \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\" (UID: \"85f6eadb-b7f4-4007-9b2c-792ec54744d1\") "
Sep 17 18:24:13.469651 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.469629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6eadb-b7f4-4007-9b2c-792ec54744d1-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "85f6eadb-b7f4-4007-9b2c-792ec54744d1" (UID: "85f6eadb-b7f4-4007-9b2c-792ec54744d1"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.470349 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.470325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6eadb-b7f4-4007-9b2c-792ec54744d1-kube-api-access-lmdmd" (OuterVolumeSpecName: "kube-api-access-lmdmd") pod "85f6eadb-b7f4-4007-9b2c-792ec54744d1" (UID: "85f6eadb-b7f4-4007-9b2c-792ec54744d1"). InnerVolumeSpecName "kube-api-access-lmdmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:13.569026 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.568998    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85f6eadb-b7f4-4007-9b2c-792ec54744d1-must-gather-output\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.569026 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.569023    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdmd\" (UniqueName: \"kubernetes.io/projected/85f6eadb-b7f4-4007-9b2c-792ec54744d1-kube-api-access-lmdmd\") on node \"ip-10-0-143-209.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.574278 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.574258    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-t9cqr_must-gather-bxzmf_85f6eadb-b7f4-4007-9b2c-792ec54744d1/copy/0.log"
Sep 17 18:24:13.574626 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.574599    2577 generic.go:356] "Generic (PLEG): container finished" podID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerID="ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f" exitCode=143
Sep 17 18:24:13.574758 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.574640    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:24:13.574758 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.574699    2577 scope.go:117] "RemoveContainer" containerID="ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f"
Sep 17 18:24:13.577060 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:24:13.577039    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxzmf\" is forbidden: User \"system:node:ip-10-0-143-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-t9cqr\": no relationship found between node 'ip-10-0-143-209.ec2.internal' and this object" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:24:13.583448 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.583428    2577 scope.go:117] "RemoveContainer" containerID="777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"
Sep 17 18:24:13.585269 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:24:13.585230    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bxzmf\" is forbidden: User \"system:node:ip-10-0-143-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-t9cqr\": no relationship found between node 'ip-10-0-143-209.ec2.internal' and this object" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" pod="openshift-must-gather-t9cqr/must-gather-bxzmf"
Sep 17 18:24:13.587020 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.586996    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" path="/var/lib/kubelet/pods/85f6eadb-b7f4-4007-9b2c-792ec54744d1/volumes"
Sep 17 18:24:13.594645 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.594632    2577 scope.go:117] "RemoveContainer" containerID="ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f"
Sep 17 18:24:13.594902 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:24:13.594884    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f\": container with ID starting with ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f not found: ID does not exist" containerID="ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f"
Sep 17 18:24:13.594948 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.594908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f"} err="failed to get container status \"ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f\": rpc error: code = NotFound desc = could not find container \"ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f\": container with ID starting with ac4e4fdc2b279ab59d745b11cd98c2a268febf6430b067be38eb30a396d3ce0f not found: ID does not exist"
Sep 17 18:24:13.594948 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.594925    2577 scope.go:117] "RemoveContainer" containerID="777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"
Sep 17 18:24:13.595190 ip-10-0-143-209 kubenswrapper[2577]: E0917 18:24:13.595171    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724\": container with ID starting with 777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724 not found: ID does not exist" containerID="777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"
Sep 17 18:24:13.595238 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:13.595199    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724"} err="failed to get container status \"777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724\": rpc error: code = NotFound desc = could not find container \"777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724\": container with ID starting with 777d24fe10f50469be9a84627843aa1f3146e6746b4f6201e8c2c59ab368c724 not found: ID does not exist"
Sep 17 18:24:15.713205 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.713176    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/alertmanager/0.log"
Sep 17 18:24:15.745087 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.745064    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/config-reloader/0.log"
Sep 17 18:24:15.763357 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.763335    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/kube-rbac-proxy-web/0.log"
Sep 17 18:24:15.782816 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.782798    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/kube-rbac-proxy/0.log"
Sep 17 18:24:15.807232 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.807211    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/kube-rbac-proxy-metric/0.log"
Sep 17 18:24:15.827905 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.827883    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/prom-label-proxy/0.log"
Sep 17 18:24:15.853085 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.853070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c1c2c4b-d5cf-455a-a27e-b75e297b303b/init-config-reloader/0.log"
Sep 17 18:24:15.909830 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:15.909800    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-w49ts_5e8a92fe-3574-4410-9565-d7926dc55b83/cluster-monitoring-operator/0.log"
Sep 17 18:24:16.046485 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.046405    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-8n59t_f3d736b9-6be9-4d09-ab3f-2c8d307aa98c/monitoring-plugin/0.log"
Sep 17 18:24:16.230080 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.230055    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nl5km_61556aed-1c07-4f91-a91d-3ead9d411694/node-exporter/0.log"
Sep 17 18:24:16.249972 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.249952    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nl5km_61556aed-1c07-4f91-a91d-3ead9d411694/kube-rbac-proxy/0.log"
Sep 17 18:24:16.269435 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.269420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nl5km_61556aed-1c07-4f91-a91d-3ead9d411694/init-textfile/0.log"
Sep 17 18:24:16.296834 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.296770    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2kszl_b654e052-4eaa-45ab-be80-4bda1f808076/kube-rbac-proxy-main/0.log"
Sep 17 18:24:16.322009 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.321991    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2kszl_b654e052-4eaa-45ab-be80-4bda1f808076/kube-rbac-proxy-self/0.log"
Sep 17 18:24:16.339861 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.339840    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2kszl_b654e052-4eaa-45ab-be80-4bda1f808076/openshift-state-metrics/0.log"
Sep 17 18:24:16.555441 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.555362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c6w6x_1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626/prometheus-operator/0.log"
Sep 17 18:24:16.575916 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.575896    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c6w6x_1e70e7bd-5fdc-44a9-9ffb-ba86c07fe626/kube-rbac-proxy/0.log"
Sep 17 18:24:16.597259 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.597238    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-px4gp_07a0b8c2-c49e-4531-b9b6-794a59f0452f/prometheus-operator-admission-webhook/0.log"
Sep 17 18:24:16.718621 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.718601    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf5dbb977-nbqmw_57d081a3-b6f5-4cd1-aa09-44eeb327809f/thanos-query/0.log"
Sep 17 18:24:16.740348 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.740327    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf5dbb977-nbqmw_57d081a3-b6f5-4cd1-aa09-44eeb327809f/kube-rbac-proxy-web/0.log"
Sep 17 18:24:16.768971 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.768948    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf5dbb977-nbqmw_57d081a3-b6f5-4cd1-aa09-44eeb327809f/kube-rbac-proxy/0.log"
Sep 17 18:24:16.799112 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.799057    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf5dbb977-nbqmw_57d081a3-b6f5-4cd1-aa09-44eeb327809f/prom-label-proxy/0.log"
Sep 17 18:24:16.828361 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.828305    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf5dbb977-nbqmw_57d081a3-b6f5-4cd1-aa09-44eeb327809f/kube-rbac-proxy-rules/0.log"
Sep 17 18:24:16.852310 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:16.852283    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf5dbb977-nbqmw_57d081a3-b6f5-4cd1-aa09-44eeb327809f/kube-rbac-proxy-metrics/0.log"
Sep 17 18:24:18.663446 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:18.663419    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/1.log"
Sep 17 18:24:18.672223 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:18.672198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cjccs_a8ef87c2-1676-4f2d-ae74-a952d8f18b14/console-operator/2.log"
Sep 17 18:24:19.122949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.122920    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86f66f5db7-nkbgs_3a77fae6-da8b-4c65-aa16-f9ece05b053d/console/0.log"
Sep 17 18:24:19.196050 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.196024    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-7bq68_01cb9b8e-1c5d-4514-831b-70a38f59b3c4/download-server/0.log"
Sep 17 18:24:19.343522 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343492    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"]
Sep 17 18:24:19.343878 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="gather"
Sep 17 18:24:19.343924 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343880    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="gather"
Sep 17 18:24:19.343924 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343889    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="copy"
Sep 17 18:24:19.343924 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343894    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="copy"
Sep 17 18:24:19.344021 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343968    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="copy"
Sep 17 18:24:19.344021 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.343977    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f6eadb-b7f4-4007-9b2c-792ec54744d1" containerName="gather"
Sep 17 18:24:19.345826 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.345812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.354701 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.354682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rcdkz\"/\"default-dockercfg-667zq\""
Sep 17 18:24:19.356984 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.356967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rcdkz\"/\"kube-root-ca.crt\""
Sep 17 18:24:19.357906 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.357892    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rcdkz\"/\"openshift-service-ca.crt\""
Sep 17 18:24:19.406757 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.406723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"]
Sep 17 18:24:19.414477 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.414453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-sys\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.414613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.414490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-lib-modules\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.414613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.414524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-podres\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.414613 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.414601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwphx\" (UniqueName: \"kubernetes.io/projected/1da8f1ff-78c6-437c-80d5-3a1f9be95449-kube-api-access-vwphx\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.414768 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.414698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-proc\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516057 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-proc\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-sys\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-lib-modules\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-podres\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-proc\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516219 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwphx\" (UniqueName: \"kubernetes.io/projected/1da8f1ff-78c6-437c-80d5-3a1f9be95449-kube-api-access-vwphx\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516429 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-sys\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516429 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-podres\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.516429 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.516281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da8f1ff-78c6-437c-80d5-3a1f9be95449-lib-modules\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.526355 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.526326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwphx\" (UniqueName: \"kubernetes.io/projected/1da8f1ff-78c6-437c-80d5-3a1f9be95449-kube-api-access-vwphx\") pod \"perf-node-gather-daemonset-p98rq\" (UID: \"1da8f1ff-78c6-437c-80d5-3a1f9be95449\") " pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.655311 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.655286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:19.925782 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:19.925756    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"]
Sep 17 18:24:19.928115 ip-10-0-143-209 kubenswrapper[2577]: W0917 18:24:19.928090    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1da8f1ff_78c6_437c_80d5_3a1f9be95449.slice/crio-68e544ae1bd4a7d61dbe82782e1ea2ac055e30027c187409501651d087625632 WatchSource:0}: Error finding container 68e544ae1bd4a7d61dbe82782e1ea2ac055e30027c187409501651d087625632: Status 404 returned error can't find the container with id 68e544ae1bd4a7d61dbe82782e1ea2ac055e30027c187409501651d087625632
Sep 17 18:24:20.606821 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.606786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq" event={"ID":"1da8f1ff-78c6-437c-80d5-3a1f9be95449","Type":"ContainerStarted","Data":"5e47fc440eb38b77483f163f58476b26fb04d4163bfe82477bd29593a501a6cc"}
Sep 17 18:24:20.606989 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.606832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq" event={"ID":"1da8f1ff-78c6-437c-80d5-3a1f9be95449","Type":"ContainerStarted","Data":"68e544ae1bd4a7d61dbe82782e1ea2ac055e30027c187409501651d087625632"}
Sep 17 18:24:20.606989 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.606921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:20.625463 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.625420    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq" podStartSLOduration=1.62540862 podStartE2EDuration="1.62540862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:20.623956023 +0000 UTC m=+1647.467119171" watchObservedRunningTime="2026-09-17 18:24:20.62540862 +0000 UTC m=+1647.468571769"
Sep 17 18:24:20.782302 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.782275    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s5qlj_ceb131a8-2b87-49d7-855e-2b2b89c9fa99/dns/0.log"
Sep 17 18:24:20.801208 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.801174    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s5qlj_ceb131a8-2b87-49d7-855e-2b2b89c9fa99/kube-rbac-proxy/0.log"
Sep 17 18:24:20.901527 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:20.901496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k6fsk_1c047389-08f6-46f9-860e-47ca8dade973/dns-node-resolver/0.log"
Sep 17 18:24:21.371749 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:21.371656    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4th94_4e097696-45f7-4f39-a75b-0d5b5b51eb7c/node-ca/0.log"
Sep 17 18:24:22.093543 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:22.093518    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cc8c845db-rbdqs_dcd70886-5382-4843-807a-b4c6b7cffa45/router/0.log"
Sep 17 18:24:22.464682 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:22.464652    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ssns9_f33d54a6-6442-4f25-84eb-aad9b3c40c0d/serve-healthcheck-canary/0.log"
Sep 17 18:24:22.914737 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:22.914705    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hf8l4_662b4cbe-fbf3-45ec-a13e-0d0b372d68cf/kube-rbac-proxy/0.log"
Sep 17 18:24:22.934910 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:22.934890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hf8l4_662b4cbe-fbf3-45ec-a13e-0d0b372d68cf/exporter/0.log"
Sep 17 18:24:22.956533 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:22.956513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hf8l4_662b4cbe-fbf3-45ec-a13e-0d0b372d68cf/extractor/0.log"
Sep 17 18:24:25.413781 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:25.413754    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-trgsr_0cfa7477-453f-4ba0-a378-98cbec28f6e6/manager/0.log"
Sep 17 18:24:25.631949 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:25.631916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qm5xs_48dc32db-3c72-4aaa-afc3-93176e9f414b/seaweedfs/0.log"
Sep 17 18:24:26.621300 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:26.621269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rcdkz/perf-node-gather-daemonset-p98rq"
Sep 17 18:24:29.713950 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:29.713923    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-lwv2h_b6e91b81-c4f3-4032-8fc9-4cdf595b52f3/migrator/0.log"
Sep 17 18:24:29.734374 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:29.734347    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-lwv2h_b6e91b81-c4f3-4032-8fc9-4cdf595b52f3/graceful-termination/0.log"
Sep 17 18:24:30.995091 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:30.995010    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:24:31.013901 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.013869    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/egress-router-binary-copy/0.log"
Sep 17 18:24:31.033175 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.033149    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/cni-plugins/0.log"
Sep 17 18:24:31.053496 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.053471    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/bond-cni-plugin/0.log"
Sep 17 18:24:31.072609 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.072591    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/routeoverride-cni/0.log"
Sep 17 18:24:31.090861 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.090844    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/whereabouts-cni-bincopy/0.log"
Sep 17 18:24:31.113229 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.113206    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-29btz_977fa5b8-3f32-4f08-98f1-e8de6c3b4927/whereabouts-cni/0.log"
Sep 17 18:24:31.449464 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.449434    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hd5cc_3b8b260d-874d-4242-88a5-7f3aa34a4126/kube-multus/0.log"
Sep 17 18:24:31.526989 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.526951    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9flf9_761dc4fd-5005-48a1-9874-bfa11455f308/network-metrics-daemon/0.log"
Sep 17 18:24:31.543151 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:31.543129    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9flf9_761dc4fd-5005-48a1-9874-bfa11455f308/kube-rbac-proxy/0.log"
Sep 17 18:24:32.374518 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.374488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/ovn-controller/0.log"
Sep 17 18:24:32.388605 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.388582    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/ovn-acl-logging/0.log"
Sep 17 18:24:32.409928 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.409902    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/kube-rbac-proxy-node/0.log"
Sep 17 18:24:32.431552 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.431530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:24:32.448992 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.448968    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/northd/0.log"
Sep 17 18:24:32.469013 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.468997    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/nbdb/0.log"
Sep 17 18:24:32.486981 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.486953    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/sbdb/0.log"
Sep 17 18:24:32.539988 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:32.539969    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlxrv_1e73f216-d440-41b0-a209-65eefbe31903/ovnkube-controller/0.log"
Sep 17 18:24:33.930347 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:33.930317    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b9jfv_de04cd13-f4a7-4f10-a014-7b98b6736e11/network-check-target-container/0.log"
Sep 17 18:24:34.876992 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:34.876966    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dcqg2_3f0b7652-f879-4ac8-a2a6-a7cf4042c0df/iptables-alerter/0.log"
Sep 17 18:24:35.531142 ip-10-0-143-209 kubenswrapper[2577]: I0917 18:24:35.531111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wt26z_f52e62dd-57e7-4a52-a6d4-bfe36454a697/tuned/0.log"