Sep 17 15:53:24.588678 ip-10-0-140-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:53:24.588689 ip-10-0-140-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:53:24.588697 ip-10-0-140-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:53:24.588837 ip-10-0-140-92 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:53:34.801591 ip-10-0-140-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 15:53:34.801612 ip-10-0-140-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6b9ad44b11f84644b846d89b7a69ae0b --
Sep 17 15:55:23.077961 ip-10-0-140-92 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:55:26.788383 ip-10-0-140-92 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:55:26.788383 ip-10-0-140-92 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:55:26.788383 ip-10-0-140-92 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:55:26.788383 ip-10-0-140-92 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:55:26.788383 ip-10-0-140-92 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:55:26.789228 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.788913    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:55:26.792706 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792690    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:26.792706 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792705    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792709    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792712    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792715    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792718    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792721    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792724    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792727    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792729    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792732    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792735    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792745    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792750    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792754    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792757    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792760    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792762    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792765    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792767    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:26.792779 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792770    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792773    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792776    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792778    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792781    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792784    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792787    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792789    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792792    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792795    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792798    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792800    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792803    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792805    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792808    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792810    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792813    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792815    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792818    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792820    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792823    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:26.793231 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792825    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792828    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792830    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792833    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792836    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792839    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792842    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792844    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792847    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792850    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792854    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792857    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792860    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792864    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792867    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792870    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792872    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792875    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792878    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:26.794076 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792881    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792884    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792887    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792890    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792893    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792895    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792898    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792900    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792903    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792905    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792908    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792911    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792913    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792916    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792918    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792921    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792923    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792926    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792929    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792932    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:26.795446 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792934    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792937    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792939    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792942    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792944    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792947    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792950    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792953    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792956    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792958    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792961    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792963    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792966    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792969    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792971    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792974    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792976    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792979    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.792983    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795097    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:26.795956 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795108    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795112    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795115    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795118    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795121    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795125    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795127    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795130    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795133    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795136    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795139    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795141    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795144    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795146    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795149    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795151    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795154    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795157    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795159    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:26.796432 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795164    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795169    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795172    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795175    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795177    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795180    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795183    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795185    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795188    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795190    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795193    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795195    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795198    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795202    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795204    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795207    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795210    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795213    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795216    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795218    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:26.796950 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795221    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795224    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795226    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795229    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795232    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795234    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795237    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795239    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795241    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795244    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795247    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795249    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795252    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795254    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795257    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795259    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795262    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795265    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795267    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:26.797471 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795271    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795275    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795278    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795280    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795283    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795286    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795289    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795292    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795295    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795298    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795300    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795303    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795305    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795308    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795311    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795313    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795316    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795318    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795321    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:26.798001 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795324    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795327    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795329    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795331    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795334    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795344    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795347    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795349    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795351    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795354    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795357    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795359    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795362    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795365    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795367    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795370    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795372    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795375    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795378    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795382    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:26.798456 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795386    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795757    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795763    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795766    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795769    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795772    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795774    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795777    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795780    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795782    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795785    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795788    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795790    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795793    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795796    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795800    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795804    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795808    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795811    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:26.798985 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795813    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795817    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795821    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795823    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795827    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795829    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795832    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795834    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795837    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795839    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795842    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795844    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795853    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795856    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795858    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795861    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795864    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795866    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795869    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:26.799451 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795872    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795874    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795876    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795879    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795882    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795884    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795887    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795889    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795892    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795894    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795897    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795899    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795902    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795905    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795907    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795910    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795912    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795915    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795917    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:26.799928 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795920    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795922    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795926    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795928    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795931    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795933    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795938    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795942    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795945    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795947    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795950    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795952    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795955    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795957    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795960    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795963    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795966    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795968    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795971    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795974    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:26.800376 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795976    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795979    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795981    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795984    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795986    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795989    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795992    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795994    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795997    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.795999    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796002    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796004    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796007    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796009    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796012    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796014    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796016    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796019    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796022    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796025    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:26.800863 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796028    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796030    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796033    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796108    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796152    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796158    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796162    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796167    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796170    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796174    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796178    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796181    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796184    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796188    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796191    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796194    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796197    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796200    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796203    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796207    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796211    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796215    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796217    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796220    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:55:26.801341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796223    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796227    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796231    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796234    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796237    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796240    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796243    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796246    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796249    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796252    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796255    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796259    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796262    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796265    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796268    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796271    2557 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796273    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796278    2557 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796281    2557 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796284    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796287    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796290    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796294    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796297    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796300    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:55:26.801940 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796303    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796306    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796309    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796312    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796316    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796318    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796321    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796324    2557 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796328    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796332    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796335    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796338    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796342    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796345    2557 flags.go:64] FLAG: --help="false"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796348    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-140-92.ec2.internal"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796351    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796354    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796356    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796359    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796363    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796366    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796368    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796371    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:55:26.802540 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796374    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796377    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796380    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796383    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796386    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796388    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796391    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796394    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796397    2557 flags.go:64] FLAG: --lock-file=""
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796400    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796402    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796405    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796410    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796414    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796416    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796419    2557 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796422    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796425    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796429    2557 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796432    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796437    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796440    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796444    2557 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796448    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796451    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:55:26.803196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796454    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796457    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796460    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796463    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796466    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796473    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796476    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796479    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796482    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796485    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796491    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796493    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796496    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796499    2557 flags.go:64] FLAG: --port="10250"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796502    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796505    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d33dac847305269"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796508    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796511    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796514    2557 flags.go:64] FLAG: --register-node="true"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796517    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796520    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796524    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796527    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796530    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796534    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:55:26.803838 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796537    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796541    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796544    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796546    2557 flags.go:64] FLAG: --runonce="false"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796549    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796553    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796556    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796559    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796562    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796565    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796568    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796571    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796573    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796576    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796579    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796582    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796585    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796602    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796605    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796610    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796613    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796616    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796620    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796623    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796626    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:55:26.804447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796629    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796632    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796635    2557 flags.go:64] FLAG: --v="2"
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796639    2557 flags.go:64] FLAG: --version="false"
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796644    2557 flags.go:64] FLAG: --vmodule=""
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796647    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.796651    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796765    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796770    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796774    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796776    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796779    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796783    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796786    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796788    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796791    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796795    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796799    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796802    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796804    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:26.805091 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796808    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796812    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796816    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796820    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796823    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796826    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796829    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796831    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796834    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796836    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796839    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796841    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796844    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796848    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796852    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796856    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796859    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796863    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796866    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796869    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:26.805578 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796873    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796875    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796878    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796881    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796885    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796888    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796891    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796894    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796896    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796899    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796902    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796905    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796908    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796910    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796913    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796915    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796918    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796920    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796923    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796926    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:26.806077 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796928    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796931    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796933    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796937    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796940    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796943    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796946    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796949    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796952    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796955    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796958    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796962    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796965    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796968    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796971    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796973    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796977    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796980    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796983    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796985    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:26.806624 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796988    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796990    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796993    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796995    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.796998    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797001    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797003    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797006    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797009    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797011    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797014    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797016    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797019    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797022    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797024    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797027    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797029    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797032    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797035    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797037    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:26.807155 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797040    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797042    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797045    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797048    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797053    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.797055    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.797060    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804003    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804018    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804036    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804046    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804100    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804104    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804108    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804111    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:26.807671 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804114    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804118    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804121    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804124    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804126    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804129    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804132    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804135    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804138    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804140    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804144    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804149    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804152    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804155    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804157    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804160    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804163    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804166    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804168    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804171    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:26.808108 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804174    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804177    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804180    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804182    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804185    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804188    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804191    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804194    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804196    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804199    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804201    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804204    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804207    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804211    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804214    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804217    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804220    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804224    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804228    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:26.808756 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804231    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804234    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804237    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804240    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804243    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804246    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804249    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804251    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804254    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804257    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804260    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804263    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804265    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804268    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804271    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804274    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804276    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804279    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804282    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804284    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:26.809289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804287    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804289    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804292    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804295    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804297    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804300    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804302    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804305    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804308    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804310    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804313    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804315    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804318    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804321    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804335    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804339    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804342    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804345    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804348    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804352    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:26.809976 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804354    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804358    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804361    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804364    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804367    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804370    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804373    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804376    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804378    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804381    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804384    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804386    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804389    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804391    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804394    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:26.810725 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804397    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804402    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804508    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804512    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804515    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804518    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804521    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804524    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804526    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804529    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804532    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804535    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804538    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804540    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804543    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:26.811143 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804546    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804549    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804552    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804557    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804561    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804567    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804572    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804577    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804581    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804586    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804605    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804608    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804611    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804614    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804617    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804620    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804623    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804625    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804628    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:26.811506 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804631    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804633    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804636    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804638    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804641    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804643    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804646    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804648    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804651    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804654    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804657    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804659    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804662    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804665    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804667    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804670    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804672    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804676    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804680    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:26.811999 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804683    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804686    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804689    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804691    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804694    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804697    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804699    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804702    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804705    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804707    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804710    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804712    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804715    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804717    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804720    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804722    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804725    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804727    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804730    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804732    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:26.812468 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804735    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804737    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804740    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804743    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804745    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804748    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804750    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804753    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804756    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804759    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804761    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804764    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804766    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804769    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804772    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804777    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804779    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804782    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804785    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804788    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:26.813056 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804790    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804793    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804795    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804798    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804801    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804804    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804808    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:26.804811    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804816    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.804908    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.806900    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.807901    2557 server.go:1019] "Starting client certificate rotation"
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.808022    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:55:26.813744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.808075    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:55:26.816041 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.810225    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:55:26.816041 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.811726    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:55:26.818309 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.818294    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:55:26.819830 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.819817    2557 log.go:25] "Validated CRI v1 image API"
Sep 17 15:55:26.821633 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.821613    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:55:26.824004 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.823985    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ae55869d-f4d1-4e53-b540-36a3e3caa3d4:/dev/nvme0n1p4 b7f978ef-70d4-41a4-931d-adb582ba45ad:/dev/nvme0n1p3]
Sep 17 15:55:26.824068 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.824003    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:55:26.828119 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.828018    2557 manager.go:216] Machine: {Timestamp:2026-09-17 15:55:26.827543095 +0000 UTC m=+2.126687586 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100323 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23574ad7792fd855db99916549fb4f SystemUUID:ec23574a-d779-2fd8-55db-99916549fb4f BootID:6b9ad44b-11f8-4644-b846-d89b7a69ae0b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:fa:b5:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:fa:b5:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:11:4e:5a:b1:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:55:26.828119 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.828115    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:55:26.828218 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.828180    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:55:26.829666 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.829647    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:55:26.829788 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.829669    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:55:26.831023 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.831013    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:55:26.831057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.831025    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:55:26.831057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.831033    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:55:26.831057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.831049    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:55:26.831141 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.831113    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:55:26.834016 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834006    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:55:26.834119 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834110    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:55:26.834159 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834140    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:55:26.834159 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834149    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:55:26.834159 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834159    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:55:26.834259 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834167    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:55:26.834259 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834176    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:55:26.834461 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834451    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:55:26.834507 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.834475    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:55:26.836825 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.836808    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 15:55:26.839352 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.839338    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:55:26.839415 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.839368    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:55:26.840462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840447    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:55:26.840506 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840468    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:55:26.840506 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840473    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:55:26.840506 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840480    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:55:26.840506 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840489    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:55:26.840506 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840494    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:55:26.840506 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840507    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840512    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840518    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840530    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840542    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840547    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840571    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:55:26.840737 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.840577    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:55:26.848971 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.848942    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:55:26.849578 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.849567    2557 server.go:1268] "Started kubelet"
Sep 17 15:55:26.849690 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.849643    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:55:26.849742 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.849684    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:55:26.849805 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.849791    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:55:26.850200 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.850166    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:55:26.850200 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.850192    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:55:26.850341 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.850286    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:55:26.850424 ip-10-0-140-92 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:55:26.884157 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.884134    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:55:26.884693 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.884580    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:55:26.885201 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.885166    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:55:26.889691 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.889670    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:55:26.889691 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.889685    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:55:26.889855 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.888816    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-92.ec2.internal.18d626c46367c5f0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-92.ec2.internal,UID:ip-10-0-140-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-92.ec2.internal,},FirstTimestamp:2026-09-17 15:55:26.849541616 +0000 UTC m=+2.148686086,LastTimestamp:2026-09-17 15:55:26.849541616 +0000 UTC m=+2.148686086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-92.ec2.internal,}"
Sep 17 15:55:26.889855 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.889776    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:26.890304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.890286    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:55:26.890304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.890287    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:55:26.890444 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.890314    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:55:26.890444 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.890402    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:55:26.890444 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.890439    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:55:26.891266 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891248    2557 factory.go:55] Registering systemd factory
Sep 17 15:55:26.891266 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891266    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:55:26.891460 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891446    2557 factory.go:153] Registering CRI-O factory
Sep 17 15:55:26.891511 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891463    2557 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:55:26.891563 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891512    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:55:26.891563 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891533    2557 factory.go:103] Registering Raw factory
Sep 17 15:55:26.891563 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891547    2557 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:55:26.891975 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.891957    2557 manager.go:349] Starting recovery of all containers
Sep 17 15:55:26.901985 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.901970    2557 manager.go:354] Recovery completed
Sep 17 15:55:26.902732 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.902705    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:55:26.902797 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.902734    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:55:26.906288 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.906277    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:26.911503 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.911479    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:26.911566 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.911523    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:26.911566 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.911549    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:26.912082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912066    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:55:26.912082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912081    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:55:26.912201 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912098    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:55:26.912201 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912172    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:26.912883 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912868    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:26.912942 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912893    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:26.912942 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.912902    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:26.913417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.913405    2557 policy_none.go:49] "None policy: Start"
Sep 17 15:55:26.913459 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.913420    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:55:26.913459 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.913431    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:55:26.914836 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.914826    2557 policy_none.go:47] "Start"
Sep 17 15:55:26.922901 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.922839    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-92.ec2.internal.18d626c467192d30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-92.ec2.internal,UID:ip-10-0-140-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-92.ec2.internal,},FirstTimestamp:2026-09-17 15:55:26.911499568 +0000 UTC m=+2.210644040,LastTimestamp:2026-09-17 15:55:26.911499568 +0000 UTC m=+2.210644040,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-92.ec2.internal,}"
Sep 17 15:55:26.934186 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.934109    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-92.ec2.internal.18d626c46719a871  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-92.ec2.internal,UID:ip-10-0-140-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-92.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-92.ec2.internal,},FirstTimestamp:2026-09-17 15:55:26.911531121 +0000 UTC m=+2.210675592,LastTimestamp:2026-09-17 15:55:26.911531121 +0000 UTC m=+2.210675592,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-92.ec2.internal,}"
Sep 17 15:55:26.947205 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.947146    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-92.ec2.internal.18d626c4671a07cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-92.ec2.internal,UID:ip-10-0-140-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-92.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-92.ec2.internal,},FirstTimestamp:2026-09-17 15:55:26.911555533 +0000 UTC m=+2.210700002,LastTimestamp:2026-09-17 15:55:26.911555533 +0000 UTC m=+2.210700002,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-92.ec2.internal,}"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.950720    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jd25"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953133    2557 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.953165    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953173    2557 server.go:87] "Starting device plugin registration server"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953402    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953424    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953517    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953607    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.953617    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.954513    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.954559    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:26.957986 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.957174    2557 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-92.ec2.internal.18d626c467192d30\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-92.ec2.internal.18d626c467192d30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-92.ec2.internal,UID:ip-10-0-140-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-92.ec2.internal,},FirstTimestamp:2026-09-17 15:55:26.911499568 +0000 UTC m=+2.210644040,LastTimestamp:2026-09-17 15:55:26.912882332 +0000 UTC m=+2.212026802,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-92.ec2.internal,}"
Sep 17 15:55:26.962418 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:26.962402    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jd25"
Sep 17 15:55:26.966734 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:26.966674    2557 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-92.ec2.internal.18d626c46719a871\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-92.ec2.internal.18d626c46719a871  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-92.ec2.internal,UID:ip-10-0-140-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-92.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-92.ec2.internal,},FirstTimestamp:2026-09-17 15:55:26.911531121 +0000 UTC m=+2.210675592,LastTimestamp:2026-09-17 15:55:26.912898001 +0000 UTC m=+2.212042470,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-92.ec2.internal,}"
Sep 17 15:55:27.020395 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.020370    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:55:27.021618 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.021600    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:55:27.021699 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.021642    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:55:27.021699 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.021660    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:55:27.021775 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.021700    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:55:27.024077 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.024061    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:27.054247 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.054210    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:27.054983 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.054966    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:27.055064 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.054994    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:27.055064 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.055003    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:27.055064 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.055023    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.066768 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.066750    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.066768 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.066769    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-92.ec2.internal\": node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.078095 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.078076    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.121776 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.121754    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"]
Sep 17 15:55:27.121864 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.121803    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:27.122534 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.122520    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:27.122633 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.122545    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:27.122633 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.122558    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:27.123925 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.123913    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:27.124044 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124030    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.124100 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124062    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:27.124548 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124533    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:27.124548 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124541    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:27.124671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124562    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:27.124671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124569    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:27.124671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124573    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:27.124671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.124579    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:27.125638 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.125626    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.125687 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.125648    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:27.126259 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.126244    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:27.126326 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.126266    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:27.126326 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.126279    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:27.154109 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.154090    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-92.ec2.internal\" not found" node="ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.158445 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.158430    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-92.ec2.internal\" not found" node="ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.179018 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.179000    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.191657 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.191635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ac68261fff0c2e6530b2d84d288ff56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal\" (UID: \"2ac68261fff0c2e6530b2d84d288ff56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.191733 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.191658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7615070c530a7fa377bebdacf1747122-config\") pod \"kube-apiserver-proxy-ip-10-0-140-92.ec2.internal\" (UID: \"7615070c530a7fa377bebdacf1747122\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.191733 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.191675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ac68261fff0c2e6530b2d84d288ff56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal\" (UID: \"2ac68261fff0c2e6530b2d84d288ff56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.279417 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.279397    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.292695 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.292673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7615070c530a7fa377bebdacf1747122-config\") pod \"kube-apiserver-proxy-ip-10-0-140-92.ec2.internal\" (UID: \"7615070c530a7fa377bebdacf1747122\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.292779 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.292702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ac68261fff0c2e6530b2d84d288ff56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal\" (UID: \"2ac68261fff0c2e6530b2d84d288ff56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.292779 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.292725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ac68261fff0c2e6530b2d84d288ff56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal\" (UID: \"2ac68261fff0c2e6530b2d84d288ff56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.292868 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.292783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7615070c530a7fa377bebdacf1747122-config\") pod \"kube-apiserver-proxy-ip-10-0-140-92.ec2.internal\" (UID: \"7615070c530a7fa377bebdacf1747122\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.292868 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.292837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ac68261fff0c2e6530b2d84d288ff56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal\" (UID: \"2ac68261fff0c2e6530b2d84d288ff56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.292942 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.292870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ac68261fff0c2e6530b2d84d288ff56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal\" (UID: \"2ac68261fff0c2e6530b2d84d288ff56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.380379 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.380327    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.457720 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.457703    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.462341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.462252    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:27.480780 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.480521    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.580991 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.580967    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.681551 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.681494    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.704387 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.704368    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:27.782049 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.782028    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.807534 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.807517    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:55:27.807906 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.807642    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:55:27.807906 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.807664    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:55:27.882486 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:27.882455    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-92.ec2.internal\" not found"
Sep 17 15:55:27.901185 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.901159    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:27.948123 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.948061    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:27.966796 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.966218    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:50:26 +0000 UTC" deadline="2028-03-17 11:57:05.085699168 +0000 UTC"
Sep 17 15:55:27.966796 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.966251    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13124h1m37.119451729s"
Sep 17 15:55:27.990125 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:27.990103    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:28.010203 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.010185    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:55:28.011255 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.011243    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal"
Sep 17 15:55:28.019183 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.019169    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:55:28.029168 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:28.029143    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac68261fff0c2e6530b2d84d288ff56.slice/crio-6fd88178215bf65e8f05ca4af7500d68e0cf9ae25082b28ca3bdb831b4448ebf WatchSource:0}: Error finding container 6fd88178215bf65e8f05ca4af7500d68e0cf9ae25082b28ca3bdb831b4448ebf: Status 404 returned error can't find the container with id 6fd88178215bf65e8f05ca4af7500d68e0cf9ae25082b28ca3bdb831b4448ebf
Sep 17 15:55:28.029739 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:28.029719    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7615070c530a7fa377bebdacf1747122.slice/crio-c59ca16d993f7595a0ca7adbdd8f7b812b6a7b84ff13bd0832fb8ae1c6bc644f WatchSource:0}: Error finding container c59ca16d993f7595a0ca7adbdd8f7b812b6a7b84ff13bd0832fb8ae1c6bc644f: Status 404 returned error can't find the container with id c59ca16d993f7595a0ca7adbdd8f7b812b6a7b84ff13bd0832fb8ae1c6bc644f
Sep 17 15:55:28.035629 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.035611    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:55:28.734445 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.734404    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:28.835428 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.835398    2557 apiserver.go:52] "Watching apiserver"
Sep 17 15:55:28.843704 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.843677    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:55:28.844282 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.844251    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wvx8p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp","openshift-image-registry/node-ca-r74cc","openshift-multus/multus-r64lr","openshift-network-diagnostics/network-check-target-m9jbf","openshift-network-operator/iptables-alerter-d99sw","kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cncn6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal","openshift-multus/multus-additional-cni-plugins-q9zbj","openshift-multus/network-metrics-daemon-6kjfr","openshift-ovn-kubernetes/ovnkube-node-bnw9m","kube-system/global-pull-secret-syncer-vqfpz"]
Sep 17 15:55:28.845903 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.845852    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.847061 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.847037    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.849015 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.848773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:28.850226 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850203    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:55:28.851032 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850414    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:55:28.851032 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850486    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:55:28.851032 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850638    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:55:28.851032 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850809    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.851032 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850890    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:55:28.851032 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.850906    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:55:28.851828 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.851611    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:55:28.852685 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.852081    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k9w7s\""
Sep 17 15:55:28.852685 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.852362    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r8m68\""
Sep 17 15:55:28.852823 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.852782    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:55:28.853033 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.852955    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:55:28.854138 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.854119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:28.854272 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:28.854231    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:28.854366 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.854303    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:55:28.854452 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.854433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fz5j4\""
Sep 17 15:55:28.854582 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.854566    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:55:28.855472 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.855453    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:28.855568 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.855488    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-76pft\""
Sep 17 15:55:28.855700 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.855682    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:55:28.857248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.857206    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.858485 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.858382    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:28.860126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.860107    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:28.860216 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:28.860194    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:28.861821 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.861783    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.862345 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.861948    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:28.862345 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862239    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:28.862480 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862451    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:55:28.862480 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862470    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:28.862580 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862523    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:28.862966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862948    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhnkc\""
Sep 17 15:55:28.862966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862960    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9m8hk\""
Sep 17 15:55:28.862966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862952    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:55:28.863127 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.863012    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:55:28.863127 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.862965    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8f7rk\""
Sep 17 15:55:28.863300 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.863281    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:28.863404 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:28.863335    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:28.870891 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.870864    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:55:28.870995 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.870895    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:55:28.870995 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.870922    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:55:28.870995 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.870966    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:55:28.871156 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.870969    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:55:28.871156 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.871080    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:55:28.871156 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.871108    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wfxkg\""
Sep 17 15:55:28.890802 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.890672    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:55:28.891626 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.891610    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:55:28.901675 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-ovnkube-config\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.901675 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901614    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-hostroot\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.901675 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-multus-certs\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.901675 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-tmp\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901688    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5829edb8-7ad1-4a80-978f-29f5f80d8140-agent-certs\") pod \"konnectivity-agent-wvx8p\" (UID: \"5829edb8-7ad1-4a80-978f-29f5f80d8140\") " pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-cni-bin\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxrl\" (UniqueName: \"kubernetes.io/projected/d550124e-4cac-4b66-a518-a08275670788-kube-api-access-ngxrl\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-run\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901887    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-sys\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.901921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjxz\" (UniqueName: \"kubernetes.io/projected/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-kube-api-access-9fjxz\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-kubelet\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-systemd-units\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.901985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb842786-bcb1-4e36-8ccf-2afbff487c28-cni-binary-copy\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysctl-conf\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-env-overrides\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05678d54-6457-4d4f-996c-72ffa691cb2b-ovn-node-metrics-cert\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902101    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-system-cni-dir\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-kubelet\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6569fbe-f209-42e5-aa02-662eef8b30a1-iptables-alerter-script\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-run-netns\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902258    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-cnibin\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902282    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysctl-d\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-lib-modules\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-sys-fs\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-k8s-cni-cncf-io\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-cni-bin\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-conf-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-daemon-config\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902454    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysconfig\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902488    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-systemd\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902532    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da2f1f6b-54eb-409d-b042-48403b5dbaf2-host\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-cni-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902602    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-cni-multus\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz88m\" (UniqueName: \"kubernetes.io/projected/eb842786-bcb1-4e36-8ccf-2afbff487c28-kube-api-access-lz88m\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-modprobe-d\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-systemd\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902739    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-var-lib-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.902817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902779    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk48v\" (UniqueName: \"kubernetes.io/projected/da2f1f6b-54eb-409d-b042-48403b5dbaf2-kube-api-access-wk48v\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tzv\" (UniqueName: \"kubernetes.io/projected/71c49110-1c75-4ad3-8118-23de3f1a05ec-kube-api-access-j6tzv\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902879    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-netns\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6569fbe-f209-42e5-aa02-662eef8b30a1-host-slash\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-var-lib-kubelet\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902960    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.902997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92mh\" (UniqueName: \"kubernetes.io/projected/e376dd58-f13e-43de-bb4f-e2cac00b537f-kube-api-access-x92mh\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-node-log\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-ovnkube-script-lib\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-socket-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-device-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903189    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da2f1f6b-54eb-409d-b042-48403b5dbaf2-serviceca\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-log-socket\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903274    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-cnibin\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.903438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-kubernetes\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903358    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-host\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903434    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5829edb8-7ad1-4a80-978f-29f5f80d8140-konnectivity-ca\") pod \"konnectivity-agent-wvx8p\" (UID: \"5829edb8-7ad1-4a80-978f-29f5f80d8140\") " pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-etc-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903527    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-cni-netd\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903585    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmhs\" (UniqueName: \"kubernetes.io/projected/05678d54-6457-4d4f-996c-72ffa691cb2b-kube-api-access-4fmhs\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903626    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-registration-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903683    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-tuned\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-slash\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-ovn\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d18cce37-76b4-406d-b113-b47a3c850d05-dbus\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:28.903894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-etc-kubernetes\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.904756 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:28.904756 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903842    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d18cce37-76b4-406d-b113-b47a3c850d05-kubelet-config\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:28.904756 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903865    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-system-cni-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.904756 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-socket-dir-parent\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:28.904756 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.903925    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45z9n\" (UniqueName: \"kubernetes.io/projected/a6569fbe-f209-42e5-aa02-662eef8b30a1-kube-api-access-45z9n\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:28.910213 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.910191    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:55:28.949660 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.949632    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zkvt7"
Sep 17 15:55:28.965221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:28.965191    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zkvt7"
Sep 17 15:55:29.004708 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-kubelet\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.004708 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-systemd-units\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.004708 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb842786-bcb1-4e36-8ccf-2afbff487c28-cni-binary-copy\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysctl-conf\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-env-overrides\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004749    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-kubelet\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-systemd-units\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004795    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05678d54-6457-4d4f-996c-72ffa691cb2b-ovn-node-metrics-cert\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-system-cni-dir\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-kubelet\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6569fbe-f209-42e5-aa02-662eef8b30a1-iptables-alerter-script\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004880    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysctl-conf\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.004927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004914    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-run-netns\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-system-cni-dir\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004935    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-kubelet\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004938    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-cnibin\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-run-netns\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-cnibin\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.004990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysctl-d\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-lib-modules\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-sys-fs\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-k8s-cni-cncf-io\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-cni-bin\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-conf-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-daemon-config\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-sys-fs\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005130    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysconfig\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005182    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-lib-modules\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysconfig\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.005420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-env-overrides\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005266    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-conf-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005269    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-k8s-cni-cncf-io\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-cni-bin\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005284    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-sysctl-d\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005317    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-systemd\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb842786-bcb1-4e36-8ccf-2afbff487c28-cni-binary-copy\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da2f1f6b-54eb-409d-b042-48403b5dbaf2-host\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005413    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-cni-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005422    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da2f1f6b-54eb-409d-b042-48403b5dbaf2-host\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-cni-multus\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz88m\" (UniqueName: \"kubernetes.io/projected/eb842786-bcb1-4e36-8ccf-2afbff487c28-kube-api-access-lz88m\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005485    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-modprobe-d\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6569fbe-f209-42e5-aa02-662eef8b30a1-iptables-alerter-script\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-systemd\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-var-lib-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk48v\" (UniqueName: \"kubernetes.io/projected/da2f1f6b-54eb-409d-b042-48403b5dbaf2-kube-api-access-wk48v\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.006256 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005607    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-cni-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005609    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tzv\" (UniqueName: \"kubernetes.io/projected/71c49110-1c75-4ad3-8118-23de3f1a05ec-kube-api-access-j6tzv\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-modprobe-d\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-var-lib-cni-multus\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.005624    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005412    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-systemd\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-netns\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005716    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-daemon-config\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-netns\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.005736    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:55:29.505694744 +0000 UTC m=+4.804839222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6569fbe-f209-42e5-aa02-662eef8b30a1-host-slash\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-var-lib-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-var-lib-kubelet\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005810    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6569fbe-f209-42e5-aa02-662eef8b30a1-host-slash\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92mh\" (UniqueName: \"kubernetes.io/projected/e376dd58-f13e-43de-bb4f-e2cac00b537f-kube-api-access-x92mh\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005886    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-systemd\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.007143 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005947    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-var-lib-kubelet\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-node-log\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.005993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-ovnkube-script-lib\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-socket-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006061    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-device-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-node-log\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da2f1f6b-54eb-409d-b042-48403b5dbaf2-serviceca\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-log-socket\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-cnibin\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006156    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-kubernetes\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-host\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006180    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-socket-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006226    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-cnibin\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.007982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-kubernetes\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-device-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5829edb8-7ad1-4a80-978f-29f5f80d8140-konnectivity-ca\") pod \"konnectivity-agent-wvx8p\" (UID: \"5829edb8-7ad1-4a80-978f-29f5f80d8140\") " pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006349    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-etc-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-etc-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-cni-netd\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmhs\" (UniqueName: \"kubernetes.io/projected/05678d54-6457-4d4f-996c-72ffa691cb2b-kube-api-access-4fmhs\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-registration-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006536    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-tuned\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006538    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-ovnkube-script-lib\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-slash\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006587    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-ovn\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.008803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d18cce37-76b4-406d-b113-b47a3c850d05-dbus\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-etc-kubernetes\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006703    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71c49110-1c75-4ad3-8118-23de3f1a05ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006745    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-openvswitch\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006752    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d18cce37-76b4-406d-b113-b47a3c850d05-kubelet-config\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-log-socket\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006777    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-system-cni-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006803    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-run-ovn\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-socket-dir-parent\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-slash\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-cni-netd\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006834    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-registration-dir\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45z9n\" (UniqueName: \"kubernetes.io/projected/a6569fbe-f209-42e5-aa02-662eef8b30a1-kube-api-access-45z9n\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006860    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-etc-kubernetes\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006863    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5829edb8-7ad1-4a80-978f-29f5f80d8140-konnectivity-ca\") pod \"konnectivity-agent-wvx8p\" (UID: \"5829edb8-7ad1-4a80-978f-29f5f80d8140\") " pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da2f1f6b-54eb-409d-b042-48403b5dbaf2-serviceca\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006878    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-ovnkube-config\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.009541 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006909    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-hostroot\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006949    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-multus-certs\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006954    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-host\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006972    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-tmp\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007004    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-system-cni-dir\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5829edb8-7ad1-4a80-978f-29f5f80d8140-agent-certs\") pod \"konnectivity-agent-wvx8p\" (UID: \"5829edb8-7ad1-4a80-978f-29f5f80d8140\") " pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.006780    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d18cce37-76b4-406d-b113-b47a3c850d05-dbus\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-host-run-multus-certs\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-multus-socket-dir-parent\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d18cce37-76b4-406d-b113-b47a3c850d05-kubelet-config\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007079    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb842786-bcb1-4e36-8ccf-2afbff487c28-hostroot\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007111    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-cni-bin\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007134    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007159    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxrl\" (UniqueName: \"kubernetes.io/projected/d550124e-4cac-4b66-a518-a08275670788-kube-api-access-ngxrl\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007214    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05678d54-6457-4d4f-996c-72ffa691cb2b-host-cni-bin\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.010250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05678d54-6457-4d4f-996c-72ffa691cb2b-ovnkube-config\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.007367    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-run\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007378    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d550124e-4cac-4b66-a518-a08275670788-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.007445    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:29.507410217 +0000 UTC m=+4.806554695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007573    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-sys\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007615    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjxz\" (UniqueName: \"kubernetes.io/projected/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-kube-api-access-9fjxz\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007641    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-sys\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007669    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-run\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.007732    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.009774    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05678d54-6457-4d4f-996c-72ffa691cb2b-ovn-node-metrics-cert\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.009833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71c49110-1c75-4ad3-8118-23de3f1a05ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.009929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5829edb8-7ad1-4a80-978f-29f5f80d8140-agent-certs\") pod \"konnectivity-agent-wvx8p\" (UID: \"5829edb8-7ad1-4a80-978f-29f5f80d8140\") " pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.010316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-etc-tuned\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.010885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.010395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-tmp\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.026314 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.026260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal" event={"ID":"2ac68261fff0c2e6530b2d84d288ff56","Type":"ContainerStarted","Data":"6fd88178215bf65e8f05ca4af7500d68e0cf9ae25082b28ca3bdb831b4448ebf"}
Sep 17 15:55:29.027456 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.027434    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal" event={"ID":"7615070c530a7fa377bebdacf1747122","Type":"ContainerStarted","Data":"c59ca16d993f7595a0ca7adbdd8f7b812b6a7b84ff13bd0832fb8ae1c6bc644f"}
Sep 17 15:55:29.039548 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.039523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxrl\" (UniqueName: \"kubernetes.io/projected/d550124e-4cac-4b66-a518-a08275670788-kube-api-access-ngxrl\") pod \"aws-ebs-csi-driver-node-7lzjp\" (UID: \"d550124e-4cac-4b66-a518-a08275670788\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.041817 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.041797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz88m\" (UniqueName: \"kubernetes.io/projected/eb842786-bcb1-4e36-8ccf-2afbff487c28-kube-api-access-lz88m\") pod \"multus-r64lr\" (UID: \"eb842786-bcb1-4e36-8ccf-2afbff487c28\") " pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.042146 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.042122    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:29.042146 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.042148    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:29.042296 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.042161    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:29.042296 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.042229    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:29.542212391 +0000 UTC m=+4.841356851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:29.044503 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.044480    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tzv\" (UniqueName: \"kubernetes.io/projected/71c49110-1c75-4ad3-8118-23de3f1a05ec-kube-api-access-j6tzv\") pod \"multus-additional-cni-plugins-q9zbj\" (UID: \"71c49110-1c75-4ad3-8118-23de3f1a05ec\") " pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.050172 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.050152    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk48v\" (UniqueName: \"kubernetes.io/projected/da2f1f6b-54eb-409d-b042-48403b5dbaf2-kube-api-access-wk48v\") pod \"node-ca-r74cc\" (UID: \"da2f1f6b-54eb-409d-b042-48403b5dbaf2\") " pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.052946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.052925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92mh\" (UniqueName: \"kubernetes.io/projected/e376dd58-f13e-43de-bb4f-e2cac00b537f-kube-api-access-x92mh\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:29.053222 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.053192    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45z9n\" (UniqueName: \"kubernetes.io/projected/a6569fbe-f209-42e5-aa02-662eef8b30a1-kube-api-access-45z9n\") pod \"iptables-alerter-d99sw\" (UID: \"a6569fbe-f209-42e5-aa02-662eef8b30a1\") " pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.053422 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.053398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmhs\" (UniqueName: \"kubernetes.io/projected/05678d54-6457-4d4f-996c-72ffa691cb2b-kube-api-access-4fmhs\") pod \"ovnkube-node-bnw9m\" (UID: \"05678d54-6457-4d4f-996c-72ffa691cb2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.059426 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.059401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjxz\" (UniqueName: \"kubernetes.io/projected/1f320a77-9ae9-4978-b870-3a63f5cbe8ba-kube-api-access-9fjxz\") pod \"tuned-cncn6\" (UID: \"1f320a77-9ae9-4978-b870-3a63f5cbe8ba\") " pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.163877 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.163817    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9zbj"
Sep 17 15:55:29.172086 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.172057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp"
Sep 17 15:55:29.179912 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.179887    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r74cc"
Sep 17 15:55:29.192661 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.192636    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r64lr"
Sep 17 15:55:29.200807 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.200782    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d99sw"
Sep 17 15:55:29.209556 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.209539    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cncn6"
Sep 17 15:55:29.215149 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.215129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:29.219784 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.219768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:29.510959 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.510932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:29.511153 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.511001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:29.511153 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.511063    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:29.511153 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.511098    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:29.511153 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.511135    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.511116792 +0000 UTC m=+5.810261257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:29.511331 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.511155    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.511145071 +0000 UTC m=+5.810289537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:29.612281 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.612249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:29.612492 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.612376    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:29.612492 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.612393    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:29.612492 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.612401    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:29.612492 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:29.612458    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.612446193 +0000 UTC m=+5.911590650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:29.869444 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.869300    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb842786_bcb1_4e36_8ccf_2afbff487c28.slice/crio-38a36e9d6258daf9236348f919a712f83e415682e00ee429368aedbd4e5ca5e5 WatchSource:0}: Error finding container 38a36e9d6258daf9236348f919a712f83e415682e00ee429368aedbd4e5ca5e5: Status 404 returned error can't find the container with id 38a36e9d6258daf9236348f919a712f83e415682e00ee429368aedbd4e5ca5e5
Sep 17 15:55:29.871289 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.871180    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6569fbe_f209_42e5_aa02_662eef8b30a1.slice/crio-bb841d297da4197d160a69cff3622d8d0a64d8aca6135624436f5b637ccf048c WatchSource:0}: Error finding container bb841d297da4197d160a69cff3622d8d0a64d8aca6135624436f5b637ccf048c: Status 404 returned error can't find the container with id bb841d297da4197d160a69cff3622d8d0a64d8aca6135624436f5b637ccf048c
Sep 17 15:55:29.873287 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.873268    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd550124e_4cac_4b66_a518_a08275670788.slice/crio-27d712b38c46a4070e606cde2736b8330abfc3f98c565dcacf99eb3ef74c71bd WatchSource:0}: Error finding container 27d712b38c46a4070e606cde2736b8330abfc3f98c565dcacf99eb3ef74c71bd: Status 404 returned error can't find the container with id 27d712b38c46a4070e606cde2736b8330abfc3f98c565dcacf99eb3ef74c71bd
Sep 17 15:55:29.875192 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.875174    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2f1f6b_54eb_409d_b042_48403b5dbaf2.slice/crio-65e030c10d4cc21bb192c5be188dc292d303b260566ef1b517bd0a594a39b4ec WatchSource:0}: Error finding container 65e030c10d4cc21bb192c5be188dc292d303b260566ef1b517bd0a594a39b4ec: Status 404 returned error can't find the container with id 65e030c10d4cc21bb192c5be188dc292d303b260566ef1b517bd0a594a39b4ec
Sep 17 15:55:29.876443 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.876219    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c49110_1c75_4ad3_8118_23de3f1a05ec.slice/crio-742a36d264f57ce7591c66101556967a13b212cafb35712e3469bea658bde1a6 WatchSource:0}: Error finding container 742a36d264f57ce7591c66101556967a13b212cafb35712e3469bea658bde1a6: Status 404 returned error can't find the container with id 742a36d264f57ce7591c66101556967a13b212cafb35712e3469bea658bde1a6
Sep 17 15:55:29.877153 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.877129    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5829edb8_7ad1_4a80_978f_29f5f80d8140.slice/crio-224b8312953733582f8910cbfe5859996a6c323247c0de45b27173a50b8772f8 WatchSource:0}: Error finding container 224b8312953733582f8910cbfe5859996a6c323247c0de45b27173a50b8772f8: Status 404 returned error can't find the container with id 224b8312953733582f8910cbfe5859996a6c323247c0de45b27173a50b8772f8
Sep 17 15:55:29.878727 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.878428    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f320a77_9ae9_4978_b870_3a63f5cbe8ba.slice/crio-4cce31fa75fc42ff4631842da44fff9e1aced8e3474b8fa227ea7dcbfabbea7f WatchSource:0}: Error finding container 4cce31fa75fc42ff4631842da44fff9e1aced8e3474b8fa227ea7dcbfabbea7f: Status 404 returned error can't find the container with id 4cce31fa75fc42ff4631842da44fff9e1aced8e3474b8fa227ea7dcbfabbea7f
Sep 17 15:55:29.879687 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:55:29.879654    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05678d54_6457_4d4f_996c_72ffa691cb2b.slice/crio-03f0ad324c4d0bbd4319590e5236dabd4990b4320c515a0fa6c5572a56b70a0f WatchSource:0}: Error finding container 03f0ad324c4d0bbd4319590e5236dabd4990b4320c515a0fa6c5572a56b70a0f: Status 404 returned error can't find the container with id 03f0ad324c4d0bbd4319590e5236dabd4990b4320c515a0fa6c5572a56b70a0f
Sep 17 15:55:29.968511 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.968481    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:50:28 +0000 UTC" deadline="2028-07-04 20:52:52.55304018 +0000 UTC"
Sep 17 15:55:29.968511 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:29.968510    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15748h57m22.58453442s"
Sep 17 15:55:30.022323 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.022296    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:30.022482 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.022432    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:30.030949 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.030922    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"03f0ad324c4d0bbd4319590e5236dabd4990b4320c515a0fa6c5572a56b70a0f"}
Sep 17 15:55:30.032006 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.031980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cncn6" event={"ID":"1f320a77-9ae9-4978-b870-3a63f5cbe8ba","Type":"ContainerStarted","Data":"4cce31fa75fc42ff4631842da44fff9e1aced8e3474b8fa227ea7dcbfabbea7f"}
Sep 17 15:55:30.033110 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.033087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wvx8p" event={"ID":"5829edb8-7ad1-4a80-978f-29f5f80d8140","Type":"ContainerStarted","Data":"224b8312953733582f8910cbfe5859996a6c323247c0de45b27173a50b8772f8"}
Sep 17 15:55:30.034251 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.034229    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerStarted","Data":"742a36d264f57ce7591c66101556967a13b212cafb35712e3469bea658bde1a6"}
Sep 17 15:55:30.035401 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.035380    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r64lr" event={"ID":"eb842786-bcb1-4e36-8ccf-2afbff487c28","Type":"ContainerStarted","Data":"38a36e9d6258daf9236348f919a712f83e415682e00ee429368aedbd4e5ca5e5"}
Sep 17 15:55:30.037395 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.037371    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal" event={"ID":"7615070c530a7fa377bebdacf1747122","Type":"ContainerStarted","Data":"7e548da517c8caf9d788c1e9ebe76f1fa43b6fd77de7c593eeabd8912e5f5569"}
Sep 17 15:55:30.038846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.038823    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r74cc" event={"ID":"da2f1f6b-54eb-409d-b042-48403b5dbaf2","Type":"ContainerStarted","Data":"65e030c10d4cc21bb192c5be188dc292d303b260566ef1b517bd0a594a39b4ec"}
Sep 17 15:55:30.040104 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.040079    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp" event={"ID":"d550124e-4cac-4b66-a518-a08275670788","Type":"ContainerStarted","Data":"27d712b38c46a4070e606cde2736b8330abfc3f98c565dcacf99eb3ef74c71bd"}
Sep 17 15:55:30.041198 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.041170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d99sw" event={"ID":"a6569fbe-f209-42e5-aa02-662eef8b30a1","Type":"ContainerStarted","Data":"bb841d297da4197d160a69cff3622d8d0a64d8aca6135624436f5b637ccf048c"}
Sep 17 15:55:30.517197 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.516897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:30.517365 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.517076    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:30.517365 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.517234    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:30.517365 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.517297    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:55:32.517273402 +0000 UTC m=+7.816417875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:30.517365 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.517350    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:30.517612 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.517400    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:32.517384989 +0000 UTC m=+7.816529460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:30.618450 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.618412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:30.618650 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.618570    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:30.618650 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.618602    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:30.618650 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.618615    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:30.618815 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:30.618674    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:32.618656633 +0000 UTC m=+7.917801103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:30.903265 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.903187    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:30.968700 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.968660    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:50:28 +0000 UTC" deadline="2028-06-15 10:50:14.460076232 +0000 UTC"
Sep 17 15:55:30.968700 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:30.968698    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15282h54m43.491382543s"
Sep 17 15:55:31.023741 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:31.023713    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:31.023911 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:31.023865    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:31.023981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:31.023931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:31.024067 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:31.024045    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:31.047843 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:31.047814    2557 generic.go:356] "Generic (PLEG): container finished" podID="2ac68261fff0c2e6530b2d84d288ff56" containerID="94a3b1dc4f1a0aa66fdbc5aae88ace870a6c0711c1279ba0c6ffeeb996092ae2" exitCode=0
Sep 17 15:55:31.048716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:31.048692    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal" event={"ID":"2ac68261fff0c2e6530b2d84d288ff56","Type":"ContainerDied","Data":"94a3b1dc4f1a0aa66fdbc5aae88ace870a6c0711c1279ba0c6ffeeb996092ae2"}
Sep 17 15:55:31.072813 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:31.072257    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-92.ec2.internal" podStartSLOduration=3.072238838 podStartE2EDuration="3.072238838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:30.07796422 +0000 UTC m=+5.377108700" watchObservedRunningTime="2026-09-17 15:55:31.072238838 +0000 UTC m=+6.371383318"
Sep 17 15:55:32.022698 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:32.022656    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:32.023142 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.022795    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:32.060822 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:32.060783    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal" event={"ID":"2ac68261fff0c2e6530b2d84d288ff56","Type":"ContainerStarted","Data":"afe68a414928042d3aabc466e35e70534aa1f1f3c1e2883dec40bcfc8cc2e822"}
Sep 17 15:55:32.535378 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:32.535341    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:32.535544 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:32.535445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:32.535638 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.535568    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:32.535727 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.535699    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:32.535791 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.535754    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:55:36.535735728 +0000 UTC m=+11.834880198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:32.535899 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.535867    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:36.535853724 +0000 UTC m=+11.834998181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:32.636529 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:32.636487    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:32.636727 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.636707    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:32.636785 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.636729    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:32.636785 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.636742    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:32.636886 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:32.636809    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:36.636791135 +0000 UTC m=+11.935935595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:33.022638 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:33.022546    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:33.022796 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:33.022679    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:33.023144 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:33.022824    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:33.023144 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:33.022918    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:34.022149 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:34.022113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:34.022334 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:34.022259    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:35.022620 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:35.022379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:35.022620 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:35.022429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:35.022620 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:35.022533    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:35.023177 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:35.022672    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:36.022785 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:36.022725    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:36.023825 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.022864    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:36.571907 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:36.571857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:36.572086 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:36.571923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:36.572086 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.572039    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:36.572086 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.572082    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:36.572316 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.572121    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:44.572099593 +0000 UTC m=+19.871244065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:36.572316 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.572141    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:55:44.572130659 +0000 UTC m=+19.871275117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:36.673314 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:36.672682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:36.673314 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.672881    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:36.673314 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.672906    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:36.673314 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.672918    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:36.673314 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:36.672975    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:44.672957372 +0000 UTC m=+19.972101836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:37.024516 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:37.023918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:37.024516 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:37.024047    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:37.024516 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:37.024368    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:37.024516 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:37.024467    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:38.022666 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:38.022629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:38.022861 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:38.022764    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:39.022880 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:39.022839    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:39.022880 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:39.022869    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:39.023347 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:39.022962    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:39.023347 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:39.023060    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:40.021924 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:40.021888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:40.022098 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:40.022008    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:41.022852 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:41.022814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:41.023274 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:41.022823    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:41.023274 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:41.022966    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:41.023274 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:41.023048    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:42.022421 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:42.022384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:42.022653 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:42.022508    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:43.022738 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:43.022700    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:43.023204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:43.022706    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:43.023204 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:43.022847    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:43.023204 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:43.022902    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:44.022870 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:44.022831    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:44.023316 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.022963    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:44.629829 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:44.629790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:44.630010 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:44.629839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:44.630010 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.629957    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:44.630010 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.629958    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:44.630165 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.630015    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:56:00.629996549 +0000 UTC m=+35.929141027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:44.630165 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.630034    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:00.630023998 +0000 UTC m=+35.929168460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:44.730798 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:44.730764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:44.730965 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.730946    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:44.731047 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.730973    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:44.731047 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.730987    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:44.731141 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:44.731054    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:00.731033678 +0000 UTC m=+36.030178153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:45.022351 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:45.022323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:45.022523 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:45.022362    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:45.022523 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:45.022454    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:45.022673 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:45.022583    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:46.022008 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:46.021975    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:46.022407 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:46.022098    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:47.023693 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.023480    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:47.023693 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:47.023604    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:47.023693 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.023662    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:47.024205 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:47.023742    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:47.090881 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.090856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r64lr" event={"ID":"eb842786-bcb1-4e36-8ccf-2afbff487c28","Type":"ContainerStarted","Data":"672bdcb82d799997f16b8835de16d40a5d72e433b51a0a0eb44d153ad2c8a587"}
Sep 17 15:55:47.092304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.092281    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r74cc" event={"ID":"da2f1f6b-54eb-409d-b042-48403b5dbaf2","Type":"ContainerStarted","Data":"ca3314c5025ca1d3744676cd0024531bcd7496199d3e0808d0cb1b6a01803a1f"}
Sep 17 15:55:47.093935 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.093908    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp" event={"ID":"d550124e-4cac-4b66-a518-a08275670788","Type":"ContainerStarted","Data":"6f1196d2fe5262a9c0be4ee176f6a58a05c45c947cc3b490b6a759a1eb41d2d4"}
Sep 17 15:55:47.095350 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.095330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"c5eb004bdd91ed4156aa2b2e73a5b8a716cf3fe32d57f0506456e5fc91ab4c06"}
Sep 17 15:55:47.096763 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.096742    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cncn6" event={"ID":"1f320a77-9ae9-4978-b870-3a63f5cbe8ba","Type":"ContainerStarted","Data":"3ce01eb161bef98be0b14e22f1fdd9f5617453ea6e5462378a161b2f2e26b96c"}
Sep 17 15:55:47.098196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.098177    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wvx8p" event={"ID":"5829edb8-7ad1-4a80-978f-29f5f80d8140","Type":"ContainerStarted","Data":"aba187f5cdcd0c3e91692ff6576083bc6a7a87ad18cbbb3632b37ca87ac32a8b"}
Sep 17 15:55:47.099464 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.099446    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerStarted","Data":"661e7d4311ea17f3081fa44da447cd6c358a977b4d5e8437e6adf48114434886"}
Sep 17 15:55:47.113091 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.113056    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-92.ec2.internal" podStartSLOduration=19.11304342 podStartE2EDuration="19.11304342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:32.081469255 +0000 UTC m=+7.380613734" watchObservedRunningTime="2026-09-17 15:55:47.11304342 +0000 UTC m=+22.412187898"
Sep 17 15:55:47.132659 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.132622    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r64lr" podStartSLOduration=3.20088288 podStartE2EDuration="20.132610492s" totalImagesPullingTime="16.931727612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:47.113156981 +0000 UTC m=+22.412301438" watchObservedRunningTime="2026-09-17 15:55:47.132610492 +0000 UTC m=+22.431754964"
Sep 17 15:55:47.163332 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.163296    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cncn6" podStartSLOduration=3.28353053 podStartE2EDuration="20.16327982s" totalImagesPullingTime="16.87974929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:47.132822253 +0000 UTC m=+22.431966732" watchObservedRunningTime="2026-09-17 15:55:47.16327982 +0000 UTC m=+22.462424299"
Sep 17 15:55:47.191416 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.191382    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wvx8p" podStartSLOduration=7.4790966260000005 podStartE2EDuration="20.191363405s" totalImagesPullingTime="12.712266779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:47.164144909 +0000 UTC m=+22.463289389" watchObservedRunningTime="2026-09-17 15:55:47.191363405 +0000 UTC m=+22.490507885"
Sep 17 15:55:47.210062 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.210016    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r74cc" podStartSLOduration=3.329742316 podStartE2EDuration="20.210001587s" totalImagesPullingTime="16.880259271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:47.209732805 +0000 UTC m=+22.508877305" watchObservedRunningTime="2026-09-17 15:55:47.210001587 +0000 UTC m=+22.509146065"
Sep 17 15:55:47.951804 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.951581    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:47.952301 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:47.952285    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:48.022790 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.022770    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:48.022886 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:48.022853    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:48.103425 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103405    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 15:55:48.103981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103758    2557 generic.go:356] "Generic (PLEG): container finished" podID="05678d54-6457-4d4f-996c-72ffa691cb2b" containerID="a4ec31a79dda3fe835f6e9730784b1fbd4a2bf2f7f509c530c48d3896a18887c" exitCode=1
Sep 17 15:55:48.103981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103825    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"3b24d1bf243c4765d479fc7440997c23e4db0eeef20ec469206c8fb7a293e631"}
Sep 17 15:55:48.103981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"4e78e43678ff1d19d0c1c2bb9a08d9b8fb506793e13ccbc86efe47a73f84f514"}
Sep 17 15:55:48.103981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"4714994332ef2aa1e446ec104573f6f6d7f391900de56c71b5a98f9191614765"}
Sep 17 15:55:48.103981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103946    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"741c6b72a4c269085962d0ed5be7aa155ad1dadc0841f0b40f2d8c4985ec37db"}
Sep 17 15:55:48.103981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.103974    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerDied","Data":"a4ec31a79dda3fe835f6e9730784b1fbd4a2bf2f7f509c530c48d3896a18887c"}
Sep 17 15:55:48.105247 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.105221    2557 generic.go:356] "Generic (PLEG): container finished" podID="71c49110-1c75-4ad3-8118-23de3f1a05ec" containerID="661e7d4311ea17f3081fa44da447cd6c358a977b4d5e8437e6adf48114434886" exitCode=0
Sep 17 15:55:48.105343 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.105249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerDied","Data":"661e7d4311ea17f3081fa44da447cd6c358a977b4d5e8437e6adf48114434886"}
Sep 17 15:55:48.106650 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.106631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d99sw" event={"ID":"a6569fbe-f209-42e5-aa02-662eef8b30a1","Type":"ContainerStarted","Data":"9ba477f307a14142f41b1e9be434533e486772b7cf77889bda4b07a091752913"}
Sep 17 15:55:48.379767 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.379741    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:55:48.965049 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.964940    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:55:48.379757925Z","UUID":"83face73-0e75-4b95-b599-18f3c7d4de26","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:55:48.966630 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.966606    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:55:48.966765 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:48.966643    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:55:49.022203 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:49.022175    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:49.022352 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:49.022310    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:49.022553 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:49.022526    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:49.022680 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:49.022659    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:49.111626 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:49.111580    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:55:49.112041 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:49.111798    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp" event={"ID":"d550124e-4cac-4b66-a518-a08275670788","Type":"ContainerStarted","Data":"a2ea1f406e3de99076ee4d9a0d40ff6f4d8ec73d5e26597618727300ce12ad22"}
Sep 17 15:55:50.021851 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:50.021770    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:50.022012 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:50.021883    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:50.120124 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:50.120091    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 15:55:50.120671 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:50.120542    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"ac5154d272684b66a2c6263fe3c304584953a024a0f0819b79b5b36879cd4efd"}
Sep 17 15:55:50.122890 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:50.122868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp" event={"ID":"d550124e-4cac-4b66-a518-a08275670788","Type":"ContainerStarted","Data":"54d1140b8b83ecc2533460fb0306e5d85e2a9466e681275bcfa1a5c4e2de0d0a"}
Sep 17 15:55:50.141649 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:50.141586    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d99sw" podStartSLOduration=6.276911169 podStartE2EDuration="23.141573807s" totalImagesPullingTime="16.864662638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:48.156869116 +0000 UTC m=+23.456013596" watchObservedRunningTime="2026-09-17 15:55:50.141573807 +0000 UTC m=+25.440718320"
Sep 17 15:55:51.021894 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:51.021862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:51.022058 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:51.021966    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:51.022058 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:51.021862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:51.022191 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:51.022152    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:52.022033 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:52.022004    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:52.022413 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:52.022100    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:53.022771 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.022560    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:53.023338 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.022613    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:53.023338 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:53.022857    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:53.023338 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:53.022920    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:53.133069 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.133043    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 15:55:53.133421 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.133398    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"00b454985da9f3d4e57419b5036a7dd5cfa2f247859fee9c5d592ba79a2dee08"}
Sep 17 15:55:53.133730 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.133707    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:53.133871 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.133783    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:53.133952 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.133890    2557 scope.go:117] "RemoveContainer" containerID="a4ec31a79dda3fe835f6e9730784b1fbd4a2bf2f7f509c530c48d3896a18887c"
Sep 17 15:55:53.135537 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.135515    2557 generic.go:356] "Generic (PLEG): container finished" podID="71c49110-1c75-4ad3-8118-23de3f1a05ec" containerID="594395de6ec89687b27ab9916a77abda0616e565d96ee600087bc235c5b05e07" exitCode=0
Sep 17 15:55:53.135661 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.135542    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerDied","Data":"594395de6ec89687b27ab9916a77abda0616e565d96ee600087bc235c5b05e07"}
Sep 17 15:55:53.150036 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.150015    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:53.150673 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.150655    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:53.172417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:53.172376    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lzjp" podStartSLOduration=6.916400002 podStartE2EDuration="26.172362354s" totalImagesPullingTime="19.255962352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:50.141711065 +0000 UTC m=+25.440855545" watchObservedRunningTime="2026-09-17 15:55:53.172362354 +0000 UTC m=+28.471506880"
Sep 17 15:55:54.022629 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.022531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:54.022771 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:54.022645    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:54.141408 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.141384    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 15:55:54.141802 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.141721    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" event={"ID":"05678d54-6457-4d4f-996c-72ffa691cb2b","Type":"ContainerStarted","Data":"533034eeb5254e86f7e64ea816213ac80aaf1c132b16d72cd73336f859fb92c3"}
Sep 17 15:55:54.141911 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.141896    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:55:54.143760 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.143728    2557 generic.go:356] "Generic (PLEG): container finished" podID="71c49110-1c75-4ad3-8118-23de3f1a05ec" containerID="de9c52b96e4939e438bf7d9043cff8919da153ca65b1dc66419a0c2d3e848f7a" exitCode=0
Sep 17 15:55:54.143851 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.143763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerDied","Data":"de9c52b96e4939e438bf7d9043cff8919da153ca65b1dc66419a0c2d3e848f7a"}
Sep 17 15:55:54.173703 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:54.173659    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" podStartSLOduration=10.276714689 podStartE2EDuration="27.173647113s" totalImagesPullingTime="16.896932424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:54.173214399 +0000 UTC m=+29.472358875" watchObservedRunningTime="2026-09-17 15:55:54.173647113 +0000 UTC m=+29.472791591"
Sep 17 15:55:55.022439 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:55.022404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:55.022616 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:55.022513    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:55.022616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:55.022563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:55.022743 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:55.022674    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:55.151330 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:55.151247    2557 generic.go:356] "Generic (PLEG): container finished" podID="71c49110-1c75-4ad3-8118-23de3f1a05ec" containerID="439bdfa68e09437964265f7c9ed96c0137716fbfe58e003df5d71ec6b717870d" exitCode=0
Sep 17 15:55:55.151772 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:55.151330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerDied","Data":"439bdfa68e09437964265f7c9ed96c0137716fbfe58e003df5d71ec6b717870d"}
Sep 17 15:55:56.022425 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:56.022394    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:56.022633 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:56.022493    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:57.023601 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:57.023552    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:57.024082 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:57.023680    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:57.024082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:57.023738    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:57.024082 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:57.023842    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:55:57.758296 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:57.758264    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:57.758478 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:57.758390    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:55:57.759109 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:57.759080    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wvx8p"
Sep 17 15:55:58.022348 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:58.022276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:55:58.022486 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:58.022375    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:55:59.022012 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:59.021978    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:55:59.022515 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:59.022109    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:55:59.022515 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:55:59.022169    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:55:59.022515 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:55:59.022279    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:00.022379 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:00.022200    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:00.022893 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.022471    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:00.165313 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:00.165272    2557 generic.go:356] "Generic (PLEG): container finished" podID="71c49110-1c75-4ad3-8118-23de3f1a05ec" containerID="cd3c7aa69e9f575a31fabc7a015e3f1227f6f86e149e59da55b168dafa76675e" exitCode=0
Sep 17 15:56:00.165458 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:00.165317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerDied","Data":"cd3c7aa69e9f575a31fabc7a015e3f1227f6f86e149e59da55b168dafa76675e"}
Sep 17 15:56:00.641208 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:00.641174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:00.641385 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:00.641218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:00.641385 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.641321    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:00.641385 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.641323    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:00.641385 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.641372    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:56:32.641359185 +0000 UTC m=+67.940503656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:00.641385 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.641384    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:32.641378823 +0000 UTC m=+67.940523280 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:00.741520 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:00.741483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:00.741729 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.741666    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:56:00.741729 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.741684    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:56:00.741729 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.741693    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:00.741851 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:00.741742    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:32.741728344 +0000 UTC m=+68.040872814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:01.022392 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:01.022368    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:01.022849 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:01.022469    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:01.023163 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:01.022961    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:01.023163 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:01.023108    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:01.169866 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:01.169830    2557 generic.go:356] "Generic (PLEG): container finished" podID="71c49110-1c75-4ad3-8118-23de3f1a05ec" containerID="a9edb7d63dd3d0f4d0f295833e11c687f3dd8d9b091d50b496626c68c0589e84" exitCode=0
Sep 17 15:56:01.170013 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:01.169907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerDied","Data":"a9edb7d63dd3d0f4d0f295833e11c687f3dd8d9b091d50b496626c68c0589e84"}
Sep 17 15:56:02.022443 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:02.022411    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:02.022862 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:02.022503    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:02.177107 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:02.177077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" event={"ID":"71c49110-1c75-4ad3-8118-23de3f1a05ec","Type":"ContainerStarted","Data":"d74854adfbd60c204ca1200a7a5b5561e801fc85f0b9ff6c6a9810e316d78526"}
Sep 17 15:56:02.216074 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:02.216024    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q9zbj" podStartSLOduration=8.868781509 podStartE2EDuration="35.216008401s" totalImagesPullingTime="26.347226892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:02.214897312 +0000 UTC m=+37.514041791" watchObservedRunningTime="2026-09-17 15:56:02.216008401 +0000 UTC m=+37.515152880"
Sep 17 15:56:03.022133 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:03.022101    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:03.022294 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:03.022194    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:03.022346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:03.022306    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:03.022444 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:03.022423    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:04.022249 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:04.022197    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:04.022641 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:04.022319    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:05.021882 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:05.021849    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:05.022071 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:05.021854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:05.022071 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:05.021948    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:05.022071 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:05.022042    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:06.022325 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:06.022297    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:06.022723 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:06.022380    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:07.023302 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:07.023275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:07.023663 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:07.023373    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:07.023663 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:07.023462    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:07.023663 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:07.023569    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:08.022080 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:08.022048    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:08.022255 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:08.022140    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:09.022653 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:09.022617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:09.023078 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:09.022659    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:09.023078 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:09.022751    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:09.023078 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:09.022878    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:10.022405 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:10.022379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:10.022661 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:10.022470    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:11.022168 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:11.022134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:11.022653 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:11.022139    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:11.022653 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:11.022266    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:11.022653 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:11.022300    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:12.021941 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:12.021915    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:12.022111 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:12.021998    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:13.022036 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:13.022002    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:13.022486 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:13.022112    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:13.022486 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:13.022161    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:13.022486 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:13.022237    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:14.022730 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:14.022691    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:14.023200 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:14.022814    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:15.022746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:15.022716    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:15.023176 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:15.022754    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:15.023176 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:15.022828    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:15.023176 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:15.022953    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:16.022420 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:16.022383    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:16.022614 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:16.022480    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:17.022763 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:17.022730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:17.023153 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:17.022866    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:17.023153 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:17.022885    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:17.023153 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:17.022978    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:18.022790 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:18.022760    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:18.023147 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:18.022848    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:19.024832 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:19.024807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:19.025252 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:19.024896    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:19.025252 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:19.024970    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:19.025252 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:19.025053    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:20.022563 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:20.022536    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:20.022735 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:20.022635    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:21.022627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:21.022578    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:21.023027 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:21.022706    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:21.023027 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:21.022759    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:21.023027 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:21.022838    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:22.022718 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:22.022681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:22.023101 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:22.022776    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:23.021969 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:23.021931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:23.022172 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:23.021932    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:23.022172 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:23.022051    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:23.022172 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:23.022142    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:24.022005 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:24.021956    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:24.022404 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:24.022065    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:25.022078 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:25.022047    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:25.022556 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:25.022048    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:25.022556 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:25.022147    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:25.022556 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:25.022218    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:25.161224 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:25.161188    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m" podUID="05678d54-6457-4d4f-996c-72ffa691cb2b" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 15:56:26.022582 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:26.022550    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:26.023034 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:26.022679    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:27.023948 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:27.023918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:27.024350 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:27.024041    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:27.024350 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:27.024086    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:27.024350 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:27.024172    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:28.022247 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:28.022217    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:28.022412 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:28.022308    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:29.025320 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:29.025281    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:29.025780 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:29.025398    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:29.025780 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:29.025444    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:29.025780 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:29.025533    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:30.022693 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:30.022663    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:30.022869 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:30.022758    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:31.024720 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:31.024688    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:31.025175 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:31.024694    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:31.025175 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:31.024796    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:31.025175 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:31.024852    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:32.022224 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:32.022194    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:32.022417 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.022300    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:32.675274 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:32.675203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:32.675274 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:32.675251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:32.675718 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.675381    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:32.675718 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.675406    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:32.675718 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.675428    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret podName:d18cce37-76b4-406d-b113-b47a3c850d05 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:36.675415673 +0000 UTC m=+131.974560130 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret") pod "global-pull-secret-syncer-vqfpz" (UID: "d18cce37-76b4-406d-b113-b47a3c850d05") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:32.675718 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.675467    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs podName:e376dd58-f13e-43de-bb4f-e2cac00b537f nodeName:}" failed. No retries permitted until 2026-09-17 15:57:36.675448704 +0000 UTC m=+131.974593177 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs") pod "network-metrics-daemon-6kjfr" (UID: "e376dd58-f13e-43de-bb4f-e2cac00b537f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:32.776522 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:32.776497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:32.776657 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.776633    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:56:32.776657 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.776648    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:56:32.776657 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.776656    2557 projected.go:196] Error preparing data for projected volume kube-api-access-v5ptw for pod openshift-network-diagnostics/network-check-target-m9jbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:32.779364 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:32.777032    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw podName:c161efdf-95e9-4b7a-9c4a-686c6af3e9a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:36.777012957 +0000 UTC m=+132.076157417 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5ptw" (UniqueName: "kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw") pod "network-check-target-m9jbf" (UID: "c161efdf-95e9-4b7a-9c4a-686c6af3e9a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:33.022925 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:33.022864    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:33.023048 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:33.022969    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:33.023048 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:33.023024    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:33.023126 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:33.023102    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:33.243395 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:33.243371    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 15:56:33.243528 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:33.243404    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb842786-bcb1-4e36-8ccf-2afbff487c28" containerID="672bdcb82d799997f16b8835de16d40a5d72e433b51a0a0eb44d153ad2c8a587" exitCode=1
Sep 17 15:56:33.243528 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:33.243437    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r64lr" event={"ID":"eb842786-bcb1-4e36-8ccf-2afbff487c28","Type":"ContainerDied","Data":"672bdcb82d799997f16b8835de16d40a5d72e433b51a0a0eb44d153ad2c8a587"}
Sep 17 15:56:33.243754 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:33.243740    2557 scope.go:117] "RemoveContainer" containerID="672bdcb82d799997f16b8835de16d40a5d72e433b51a0a0eb44d153ad2c8a587"
Sep 17 15:56:34.022308 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:34.022142    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:34.022680 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:34.022373    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:34.247065 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:34.247041    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 15:56:34.247221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:34.247093    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r64lr" event={"ID":"eb842786-bcb1-4e36-8ccf-2afbff487c28","Type":"ContainerStarted","Data":"67fe851d1d6b91eb637a9d3e71570ee202dd3971229b448ebfbed08db44a8c5f"}
Sep 17 15:56:35.022646 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:35.022613    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:35.023010 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:35.022620    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:35.023010 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:35.022710    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:35.023010 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:35.022766    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:36.022706 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:36.022674    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:36.023081 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:36.022771    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:37.025020 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:37.024990    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:37.025393 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:37.024991    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:37.025393 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:37.025077    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:37.025393 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:37.025156    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:38.022512 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:38.022480    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:38.022677 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:38.022576    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:39.022095 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:39.022066    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:39.022480 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:39.022069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:39.022480 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:39.022172    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:39.022480 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:39.022242    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:40.022553 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:40.022512    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:40.023013 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:40.022630    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:41.022349 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:41.022319    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:41.022529 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:41.022421    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:41.022529 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:41.022498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:41.022934 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:41.022627    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:42.022471 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:42.022441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:42.022677 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:42.022531    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:43.022173 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:43.022141    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:43.022349 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:43.022149    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:43.022349 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:43.022301    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:43.022435 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:43.022403    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:44.022133 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:44.022094    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:44.022521 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:44.022195    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:45.021845 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:45.021807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:45.022024 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:45.021919    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:45.022024 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:45.021999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:45.022124 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:45.022103    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:46.022310 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:46.022271    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:46.022729 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:46.022376    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:47.022757 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:47.022728    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:47.023217 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:47.022818    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:47.023217 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:47.022847    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:47.023217 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:47.022927    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:48.022331 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:48.022304    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:48.022500 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:48.022393    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:49.021906 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.021874    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:49.022569 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:49.021990    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:49.022569 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.022040    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:49.022569 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:49.022130    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:49.477934 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.477875    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kjfr"]
Sep 17 15:56:49.478112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.477994    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:49.478180 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:49.478106    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:49.481101 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.481054    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9jbf"]
Sep 17 15:56:49.481243 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.481156    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:49.481309 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:49.481250    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:49.496610 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.496561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqfpz"]
Sep 17 15:56:49.496782 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:49.496678    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:49.496782 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:49.496769    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:51.022466 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:51.022424    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:51.022466 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:51.022468    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:51.023165 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:51.022428    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:51.023165 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:51.022537    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jbf" podUID="c161efdf-95e9-4b7a-9c4a-686c6af3e9a3"
Sep 17 15:56:51.023165 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:51.022660    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kjfr" podUID="e376dd58-f13e-43de-bb4f-e2cac00b537f"
Sep 17 15:56:51.023165 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:51.022755    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfpz" podUID="d18cce37-76b4-406d-b113-b47a3c850d05"
Sep 17 15:56:52.968972 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:52.968801    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-92.ec2.internal" event="NodeReady"
Sep 17 15:56:52.969325 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:52.969065    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:56:53.022471 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.022448    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:56:53.022623 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.022458    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:56:53.022687 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.022664    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:56:53.030734 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.030715    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:56:53.031002 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.030989    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fvxq9\""
Sep 17 15:56:53.031098 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.031070    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:56:53.031149 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.031128    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:56:53.031207 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.031188    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v87nd\""
Sep 17 15:56:53.032062 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.032038    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:56:53.062946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.062907    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c8fb8969-jdnhs"]
Sep 17 15:56:53.065709 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.065694    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.078815 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.078800    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g5p99\""
Sep 17 15:56:53.078901 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.078801    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:56:53.079624 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.079607    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:56:53.085906 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.085885    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:56:53.086095 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.086081    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:56:53.086169 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.086101    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62qjw"]
Sep 17 15:56:53.088605 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.088578    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.096965 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.096946    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:56:53.097460 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.097443    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxj9s\""
Sep 17 15:56:53.097543 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.097445    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:56:53.101868 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.101854    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:56:53.111043 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.111025    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c6q8r"]
Sep 17 15:56:53.113660 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.113647    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.121843 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.121823    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62qjw"]
Sep 17 15:56:53.122296 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.122279    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8fb8969-jdnhs"]
Sep 17 15:56:53.122559 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.122544    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:56:53.122828 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.122812    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xrk56\""
Sep 17 15:56:53.123194 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.123174    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:56:53.123283 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.123180    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:56:53.123283 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.123256    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:56:53.136707 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.136688    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c6q8r"]
Sep 17 15:56:53.221416 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-image-registry-private-configuration\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221416 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlwp\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-kube-api-access-8hlwp\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221436    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/017d8d03-3f22-469b-bebe-58c6258404ac-ca-trust-extracted\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221466    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-bound-sa-token\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248km\" (UniqueName: \"kubernetes.io/projected/661ed124-a6a6-4713-a554-6e75b33d9221-kube-api-access-248km\") pod \"ingress-canary-62qjw\" (UID: \"661ed124-a6a6-4713-a554-6e75b33d9221\") " pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221539    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwclb\" (UniqueName: \"kubernetes.io/projected/4024cb32-d700-42bf-9bd5-6217de9aeea6-kube-api-access-rwclb\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/661ed124-a6a6-4713-a554-6e75b33d9221-cert\") pod \"ingress-canary-62qjw\" (UID: \"661ed124-a6a6-4713-a554-6e75b33d9221\") " pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.221615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221606    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-trusted-ca\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221963 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221629    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-installation-pull-secrets\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.221963 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221656    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.221963 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221672    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4024cb32-d700-42bf-9bd5-6217de9aeea6-config-volume\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.221963 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4024cb32-d700-42bf-9bd5-6217de9aeea6-tmp-dir\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.221963 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.221714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-registry-certificates\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322272 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322243    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-image-registry-private-configuration\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlwp\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-kube-api-access-8hlwp\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/017d8d03-3f22-469b-bebe-58c6258404ac-ca-trust-extracted\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322614 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.322422    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:56:53.322614 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.322444    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8fb8969-jdnhs: secret "image-registry-tls" not found
Sep 17 15:56:53.322614 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.322498    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls podName:017d8d03-3f22-469b-bebe-58c6258404ac nodeName:}" failed. No retries permitted until 2026-09-17 15:56:53.822479241 +0000 UTC m=+89.121623703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls") pod "image-registry-69c8fb8969-jdnhs" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac") : secret "image-registry-tls" not found
Sep 17 15:56:53.322614 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-bound-sa-token\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322614 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248km\" (UniqueName: \"kubernetes.io/projected/661ed124-a6a6-4713-a554-6e75b33d9221-kube-api-access-248km\") pod \"ingress-canary-62qjw\" (UID: \"661ed124-a6a6-4713-a554-6e75b33d9221\") " pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.322614 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwclb\" (UniqueName: \"kubernetes.io/projected/4024cb32-d700-42bf-9bd5-6217de9aeea6-kube-api-access-rwclb\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.322887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322632    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/661ed124-a6a6-4713-a554-6e75b33d9221-cert\") pod \"ingress-canary-62qjw\" (UID: \"661ed124-a6a6-4713-a554-6e75b33d9221\") " pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.322887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/017d8d03-3f22-469b-bebe-58c6258404ac-ca-trust-extracted\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322762    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-trusted-ca\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-installation-pull-secrets\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.322887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.322887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4024cb32-d700-42bf-9bd5-6217de9aeea6-config-volume\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.323169 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4024cb32-d700-42bf-9bd5-6217de9aeea6-tmp-dir\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.323169 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.322941    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-registry-certificates\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.323169 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.322960    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:53.323169 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.323009    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls podName:4024cb32-d700-42bf-9bd5-6217de9aeea6 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:53.822993706 +0000 UTC m=+89.122138181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls") pod "dns-default-c6q8r" (UID: "4024cb32-d700-42bf-9bd5-6217de9aeea6") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:53.323370 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.323347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4024cb32-d700-42bf-9bd5-6217de9aeea6-tmp-dir\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.323583 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.323543    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-registry-certificates\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.323719 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.323621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4024cb32-d700-42bf-9bd5-6217de9aeea6-config-volume\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.323923 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.323900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-trusted-ca\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.326134 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.326116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/661ed124-a6a6-4713-a554-6e75b33d9221-cert\") pod \"ingress-canary-62qjw\" (UID: \"661ed124-a6a6-4713-a554-6e75b33d9221\") " pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.326227 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.326139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-installation-pull-secrets\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.326227 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.326208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-image-registry-private-configuration\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.336803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.336776    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-248km\" (UniqueName: \"kubernetes.io/projected/661ed124-a6a6-4713-a554-6e75b33d9221-kube-api-access-248km\") pod \"ingress-canary-62qjw\" (UID: \"661ed124-a6a6-4713-a554-6e75b33d9221\") " pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.336968 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.336952    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlwp\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-kube-api-access-8hlwp\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.353233 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.353212    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwclb\" (UniqueName: \"kubernetes.io/projected/4024cb32-d700-42bf-9bd5-6217de9aeea6-kube-api-access-rwclb\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.359169 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.359150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-bound-sa-token\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.393618 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.393571    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-2dzfn"]
Sep 17 15:56:53.396524 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.396502    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62qjw"
Sep 17 15:56:53.397494 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.397476    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.403292 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.403271    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wzwjh\""
Sep 17 15:56:53.524306 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.524246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/356ce5ee-7b00-405b-975d-d63f02d9d071-hosts-file\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.524306 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.524277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356ce5ee-7b00-405b-975d-d63f02d9d071-tmp-dir\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.524463 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.524312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f84q\" (UniqueName: \"kubernetes.io/projected/356ce5ee-7b00-405b-975d-d63f02d9d071-kube-api-access-8f84q\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.625226 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.625200    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/356ce5ee-7b00-405b-975d-d63f02d9d071-hosts-file\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.625372 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.625230    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356ce5ee-7b00-405b-975d-d63f02d9d071-tmp-dir\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.625372 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.625266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f84q\" (UniqueName: \"kubernetes.io/projected/356ce5ee-7b00-405b-975d-d63f02d9d071-kube-api-access-8f84q\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.625372 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.625340    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/356ce5ee-7b00-405b-975d-d63f02d9d071-hosts-file\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.625625 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.625586    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356ce5ee-7b00-405b-975d-d63f02d9d071-tmp-dir\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.643945 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.643920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f84q\" (UniqueName: \"kubernetes.io/projected/356ce5ee-7b00-405b-975d-d63f02d9d071-kube-api-access-8f84q\") pod \"node-resolver-2dzfn\" (UID: \"356ce5ee-7b00-405b-975d-d63f02d9d071\") " pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.684473 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.684451    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62qjw"]
Sep 17 15:56:53.688367 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:56:53.688345    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661ed124_a6a6_4713_a554_6e75b33d9221.slice/crio-e786262701851c893ceff1635897ead44db28f532c9511634945622689bad4dd WatchSource:0}: Error finding container e786262701851c893ceff1635897ead44db28f532c9511634945622689bad4dd: Status 404 returned error can't find the container with id e786262701851c893ceff1635897ead44db28f532c9511634945622689bad4dd
Sep 17 15:56:53.707297 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.707280    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2dzfn"
Sep 17 15:56:53.719129 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:56:53.719108    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356ce5ee_7b00_405b_975d_d63f02d9d071.slice/crio-f4e1beeef711278d8f565afba35334c333454e0031650e4cfc11d4c6ea1075e2 WatchSource:0}: Error finding container f4e1beeef711278d8f565afba35334c333454e0031650e4cfc11d4c6ea1075e2: Status 404 returned error can't find the container with id f4e1beeef711278d8f565afba35334c333454e0031650e4cfc11d4c6ea1075e2
Sep 17 15:56:53.826429 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.826407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:53.826524 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:53.826459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:53.826569 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.826540    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:53.826569 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.826546    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:56:53.826569 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.826560    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8fb8969-jdnhs: secret "image-registry-tls" not found
Sep 17 15:56:53.826700 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.826582    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls podName:4024cb32-d700-42bf-9bd5-6217de9aeea6 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:54.826569207 +0000 UTC m=+90.125713678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls") pod "dns-default-c6q8r" (UID: "4024cb32-d700-42bf-9bd5-6217de9aeea6") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:53.826700 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:53.826613    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls podName:017d8d03-3f22-469b-bebe-58c6258404ac nodeName:}" failed. No retries permitted until 2026-09-17 15:56:54.826607467 +0000 UTC m=+90.125751923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls") pod "image-registry-69c8fb8969-jdnhs" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac") : secret "image-registry-tls" not found
Sep 17 15:56:54.293796 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:54.293754    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2dzfn" event={"ID":"356ce5ee-7b00-405b-975d-d63f02d9d071","Type":"ContainerStarted","Data":"eb6a63847dff00519845f1137c30ca393efe8530a33f8432b4d110df3294ec03"}
Sep 17 15:56:54.294214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:54.293826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2dzfn" event={"ID":"356ce5ee-7b00-405b-975d-d63f02d9d071","Type":"ContainerStarted","Data":"f4e1beeef711278d8f565afba35334c333454e0031650e4cfc11d4c6ea1075e2"}
Sep 17 15:56:54.295245 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:54.295216    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62qjw" event={"ID":"661ed124-a6a6-4713-a554-6e75b33d9221","Type":"ContainerStarted","Data":"e786262701851c893ceff1635897ead44db28f532c9511634945622689bad4dd"}
Sep 17 15:56:54.318723 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:54.318672    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2dzfn" podStartSLOduration=1.3186541809999999 podStartE2EDuration="1.318654181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:54.316566514 +0000 UTC m=+89.615711031" watchObservedRunningTime="2026-09-17 15:56:54.318654181 +0000 UTC m=+89.617798667"
Sep 17 15:56:54.833480 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:54.833440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:54.833667 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:54.833515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:54.833667 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:54.833614    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:56:54.833667 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:54.833627    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:54.833667 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:54.833631    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8fb8969-jdnhs: secret "image-registry-tls" not found
Sep 17 15:56:54.833855 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:54.833693    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls podName:4024cb32-d700-42bf-9bd5-6217de9aeea6 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:56.833674611 +0000 UTC m=+92.132819070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls") pod "dns-default-c6q8r" (UID: "4024cb32-d700-42bf-9bd5-6217de9aeea6") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:54.833855 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:54.833713    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls podName:017d8d03-3f22-469b-bebe-58c6258404ac nodeName:}" failed. No retries permitted until 2026-09-17 15:56:56.833705963 +0000 UTC m=+92.132850423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls") pod "image-registry-69c8fb8969-jdnhs" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac") : secret "image-registry-tls" not found
Sep 17 15:56:55.161126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:55.161099    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnw9m"
Sep 17 15:56:56.300847 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:56.300767    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62qjw" event={"ID":"661ed124-a6a6-4713-a554-6e75b33d9221","Type":"ContainerStarted","Data":"e864a54177c40038313bdf5fe7eefb739e58fe83fe07e6d8c4a74cd77c7913fe"}
Sep 17 15:56:56.317946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:56.317902    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62qjw" podStartSLOduration=1.055848553 podStartE2EDuration="3.317885791s" totalImagesPullingTime="2.262037238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:56.317422213 +0000 UTC m=+91.616566707" watchObservedRunningTime="2026-09-17 15:56:56.317885791 +0000 UTC m=+91.617030270"
Sep 17 15:56:56.846887 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:56.846857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:56:56.847056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:56:56.846917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:56:56.847056 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:56.847015    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:56.847056 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:56.847020    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:56:56.847056 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:56.847036    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8fb8969-jdnhs: secret "image-registry-tls" not found
Sep 17 15:56:56.847185 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:56.847077    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls podName:4024cb32-d700-42bf-9bd5-6217de9aeea6 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:00.84706053 +0000 UTC m=+96.146204988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls") pod "dns-default-c6q8r" (UID: "4024cb32-d700-42bf-9bd5-6217de9aeea6") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:56.847185 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:56:56.847090    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls podName:017d8d03-3f22-469b-bebe-58c6258404ac nodeName:}" failed. No retries permitted until 2026-09-17 15:57:00.847084852 +0000 UTC m=+96.146229309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls") pod "image-registry-69c8fb8969-jdnhs" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac") : secret "image-registry-tls" not found
Sep 17 15:57:00.878387 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:00.878339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:00.878924 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:00.878413    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:00.878924 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:00.878518    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:57:00.878924 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:00.878521    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:57:00.878924 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:00.878529    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8fb8969-jdnhs: secret "image-registry-tls" not found
Sep 17 15:57:00.878924 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:00.878624    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls podName:4024cb32-d700-42bf-9bd5-6217de9aeea6 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:08.87858066 +0000 UTC m=+104.177725122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls") pod "dns-default-c6q8r" (UID: "4024cb32-d700-42bf-9bd5-6217de9aeea6") : secret "dns-default-metrics-tls" not found
Sep 17 15:57:00.878924 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:00.878654    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls podName:017d8d03-3f22-469b-bebe-58c6258404ac nodeName:}" failed. No retries permitted until 2026-09-17 15:57:08.878643312 +0000 UTC m=+104.177787784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls") pod "image-registry-69c8fb8969-jdnhs" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac") : secret "image-registry-tls" not found
Sep 17 15:57:07.623875 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.623836    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-42tk6"]
Sep 17 15:57:07.626785 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.626768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.630579 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.630557    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:57:07.637189 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.637160    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:57:07.638912 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.638898    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5b2qd\""
Sep 17 15:57:07.640739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.640722    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:57:07.641077 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.641064    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:57:07.648450 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.648422    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42tk6"]
Sep 17 15:57:07.721901 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.721872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.722082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.721937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9097059a-4324-4b3a-880c-ffb932e01953-crio-socket\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.722082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.722001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9097059a-4324-4b3a-880c-ffb932e01953-data-volume\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.722082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.722079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.722193 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.722100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlbd2\" (UniqueName: \"kubernetes.io/projected/9097059a-4324-4b3a-880c-ffb932e01953-kube-api-access-jlbd2\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823125 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9097059a-4324-4b3a-880c-ffb932e01953-crio-socket\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9097059a-4324-4b3a-880c-ffb932e01953-data-volume\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823248 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:07.823220    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:07.823388 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823388 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:07.823298    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:08.323279319 +0000 UTC m=+103.622423796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:07.823388 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlbd2\" (UniqueName: \"kubernetes.io/projected/9097059a-4324-4b3a-880c-ffb932e01953-kube-api-access-jlbd2\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823388 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9097059a-4324-4b3a-880c-ffb932e01953-crio-socket\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.823388 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:07.823362    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:07.823549 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:07.823406    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:08.323395754 +0000 UTC m=+103.622540212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:07.823648 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.823631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9097059a-4324-4b3a-880c-ffb932e01953-data-volume\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:07.838212 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:07.838192    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlbd2\" (UniqueName: \"kubernetes.io/projected/9097059a-4324-4b3a-880c-ffb932e01953-kube-api-access-jlbd2\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:08.327874 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:08.327838    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:08.328050 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.327974    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:08.328050 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:08.328013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:08.328199 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.328062    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:09.32803894 +0000 UTC m=+104.627183413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:08.328199 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.328113    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:08.328199 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.328179    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:09.328164729 +0000 UTC m=+104.627309189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:08.933383 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:08.933344    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:08.933870 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.933508    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:57:08.933870 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:08.933518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:08.933870 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.933578    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls podName:4024cb32-d700-42bf-9bd5-6217de9aeea6 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:24.933559997 +0000 UTC m=+120.232704455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls") pod "dns-default-c6q8r" (UID: "4024cb32-d700-42bf-9bd5-6217de9aeea6") : secret "dns-default-metrics-tls" not found
Sep 17 15:57:08.933870 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.933642    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:57:08.933870 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.933654    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8fb8969-jdnhs: secret "image-registry-tls" not found
Sep 17 15:57:08.933870 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:08.933702    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls podName:017d8d03-3f22-469b-bebe-58c6258404ac nodeName:}" failed. No retries permitted until 2026-09-17 15:57:24.933691418 +0000 UTC m=+120.232835889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls") pod "image-registry-69c8fb8969-jdnhs" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac") : secret "image-registry-tls" not found
Sep 17 15:57:09.336957 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:09.336873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:09.337112 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:09.336985    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:09.337112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:09.336990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:09.337112 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:09.337039    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:11.337023016 +0000 UTC m=+106.636167501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:09.337112 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:09.337106    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:09.337300 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:09.337163    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:11.337148152 +0000 UTC m=+106.636292609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:09.355096 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:09.355073    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2dzfn_356ce5ee-7b00-405b-975d-d63f02d9d071/dns-node-resolver/0.log"
Sep 17 15:57:10.560298 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:10.560269    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r74cc_da2f1f6b-54eb-409d-b042-48403b5dbaf2/node-ca/0.log"
Sep 17 15:57:11.156785 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:11.156758    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62qjw_661ed124-a6a6-4713-a554-6e75b33d9221/serve-healthcheck-canary/0.log"
Sep 17 15:57:11.351190 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:11.351160    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:11.351382 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:11.351202    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:11.351382 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:11.351304    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:11.351382 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:11.351309    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:11.351382 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:11.351357    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:15.35134284 +0000 UTC m=+110.650487311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:11.351382 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:11.351370    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:15.35136369 +0000 UTC m=+110.650508148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:15.377308 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:15.377268    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:15.377708 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:15.377326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:15.377708 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:15.377434    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:15.377708 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:15.377452    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:15.377708 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:15.377498    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:23.377480339 +0000 UTC m=+118.676624798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:15.377708 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:15.377513    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm podName:9097059a-4324-4b3a-880c-ffb932e01953 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:23.377505923 +0000 UTC m=+118.676650380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tk6" (UID: "9097059a-4324-4b3a-880c-ffb932e01953") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:23.431700 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:23.431664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:23.432139 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:23.431710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:23.432243 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:23.432214    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9097059a-4324-4b3a-880c-ffb932e01953-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:23.433854 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:23.433834    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9097059a-4324-4b3a-880c-ffb932e01953-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tk6\" (UID: \"9097059a-4324-4b3a-880c-ffb932e01953\") " pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:23.535010 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:23.534979    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42tk6"
Sep 17 15:57:23.842538 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:23.842463    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42tk6"]
Sep 17 15:57:23.851311 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:23.851281    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9097059a_4324_4b3a_880c_ffb932e01953.slice/crio-ae2770a2f4d36f14193df7a0912e416af1494b2ab901d6654de409ccc013d045 WatchSource:0}: Error finding container ae2770a2f4d36f14193df7a0912e416af1494b2ab901d6654de409ccc013d045: Status 404 returned error can't find the container with id ae2770a2f4d36f14193df7a0912e416af1494b2ab901d6654de409ccc013d045
Sep 17 15:57:24.368528 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:24.368486    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tk6" event={"ID":"9097059a-4324-4b3a-880c-ffb932e01953","Type":"ContainerStarted","Data":"94f68fbecbfd628f60f5156eae119a372661d703e33a02fb5604532fb1279f0f"}
Sep 17 15:57:24.368741 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:24.368556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tk6" event={"ID":"9097059a-4324-4b3a-880c-ffb932e01953","Type":"ContainerStarted","Data":"ae2770a2f4d36f14193df7a0912e416af1494b2ab901d6654de409ccc013d045"}
Sep 17 15:57:24.943682 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:24.943641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:24.944090 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:24.943712    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:24.946588 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:24.946561    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4024cb32-d700-42bf-9bd5-6217de9aeea6-metrics-tls\") pod \"dns-default-c6q8r\" (UID: \"4024cb32-d700-42bf-9bd5-6217de9aeea6\") " pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:24.946750 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:24.946729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"image-registry-69c8fb8969-jdnhs\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") " pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:25.173187 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:25.173155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:25.221114 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:25.221030    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:25.373970 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:25.373929    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tk6" event={"ID":"9097059a-4324-4b3a-880c-ffb932e01953","Type":"ContainerStarted","Data":"fec1739c86239eff0ee467c91808c544ebd2923004309f3c20e808f8c019b3e0"}
Sep 17 15:57:25.462125 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:25.462091    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8fb8969-jdnhs"]
Sep 17 15:57:25.512784 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:25.512717    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c6q8r"]
Sep 17 15:57:25.687510 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:25.687473    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017d8d03_3f22_469b_bebe_58c6258404ac.slice/crio-d4e81c125a6c689b0117da83ab446e71dbd697bb2324872300ec5b52cca26f8f WatchSource:0}: Error finding container d4e81c125a6c689b0117da83ab446e71dbd697bb2324872300ec5b52cca26f8f: Status 404 returned error can't find the container with id d4e81c125a6c689b0117da83ab446e71dbd697bb2324872300ec5b52cca26f8f
Sep 17 15:57:25.688266 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:25.687960    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4024cb32_d700_42bf_9bd5_6217de9aeea6.slice/crio-cef7dfd08cb245d37ee0d7f534bd945c66b9a6254c81a3054ceeedc0a6cb9595 WatchSource:0}: Error finding container cef7dfd08cb245d37ee0d7f534bd945c66b9a6254c81a3054ceeedc0a6cb9595: Status 404 returned error can't find the container with id cef7dfd08cb245d37ee0d7f534bd945c66b9a6254c81a3054ceeedc0a6cb9595
Sep 17 15:57:26.378391 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.378347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6q8r" event={"ID":"4024cb32-d700-42bf-9bd5-6217de9aeea6","Type":"ContainerStarted","Data":"cef7dfd08cb245d37ee0d7f534bd945c66b9a6254c81a3054ceeedc0a6cb9595"}
Sep 17 15:57:26.380190 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.380159    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" event={"ID":"017d8d03-3f22-469b-bebe-58c6258404ac","Type":"ContainerStarted","Data":"9e500b752b9c3d5dfc994c81fab132c637ee087b043724c00bbb0c0b277b1a65"}
Sep 17 15:57:26.380339 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.380218    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" event={"ID":"017d8d03-3f22-469b-bebe-58c6258404ac","Type":"ContainerStarted","Data":"d4e81c125a6c689b0117da83ab446e71dbd697bb2324872300ec5b52cca26f8f"}
Sep 17 15:57:26.380339 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.380246    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:26.382635 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.382611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tk6" event={"ID":"9097059a-4324-4b3a-880c-ffb932e01953","Type":"ContainerStarted","Data":"c6d13eb9a8f7c61424fae88827e2b10474f36b6fd55cc0febd827cec5bf0e97f"}
Sep 17 15:57:26.401858 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.401407    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" podStartSLOduration=116.40139194 podStartE2EDuration="1m56.40139194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:26.400548866 +0000 UTC m=+121.699693346" watchObservedRunningTime="2026-09-17 15:57:26.40139194 +0000 UTC m=+121.700536421"
Sep 17 15:57:26.417154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:26.417107    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-42tk6" podStartSLOduration=17.640255039 podStartE2EDuration="19.41708953s" totalImagesPullingTime="1.776834491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:57:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:26.41668396 +0000 UTC m=+121.715828461" watchObservedRunningTime="2026-09-17 15:57:26.41708953 +0000 UTC m=+121.716234010"
Sep 17 15:57:27.387520 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:27.387483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6q8r" event={"ID":"4024cb32-d700-42bf-9bd5-6217de9aeea6","Type":"ContainerStarted","Data":"68f0a71ebb056847c84070c54dd2eb3ccd23e2f93b4be42790c145e956dc41cc"}
Sep 17 15:57:27.387919 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:27.387579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6q8r" event={"ID":"4024cb32-d700-42bf-9bd5-6217de9aeea6","Type":"ContainerStarted","Data":"c18a92c584dee9276a1c43657a111b6e291ea5e02a547296df92e677470bd4a1"}
Sep 17 15:57:27.387919 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:27.387634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:27.424546 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:27.424493    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c6q8r" podStartSLOduration=32.925068073 podStartE2EDuration="34.424479465s" totalImagesPullingTime="1.499411392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:27.424278258 +0000 UTC m=+122.723422758" watchObservedRunningTime="2026-09-17 15:57:27.424479465 +0000 UTC m=+122.723623943"
Sep 17 15:57:32.859370 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.859338    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"]
Sep 17 15:57:32.863715 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.863699    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-xxgdj"]
Sep 17 15:57:32.863832 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.863814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:32.866377 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.866358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:32.866377 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.866366    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 15:57:32.866517 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.866428    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gwsjt\""
Sep 17 15:57:32.868401 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.868382    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-msvq6\""
Sep 17 15:57:32.869001 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.868965    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:57:32.869084 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.869069    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:57:32.873216 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.873197    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"]
Sep 17 15:57:32.874122 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.874096    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-xxgdj"]
Sep 17 15:57:32.883288 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.883267    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8fb8969-jdnhs"]
Sep 17 15:57:32.903877 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.903849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4794d008-e662-4299-8e12-b4b736b8b915-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-6l7tx\" (UID: \"4794d008-e662-4299-8e12-b4b736b8b915\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:32.903971 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:32.903889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679f5\" (UniqueName: \"kubernetes.io/projected/d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23-kube-api-access-679f5\") pod \"downloads-6594f4c75-xxgdj\" (UID: \"d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23\") " pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:33.004440 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.004419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4794d008-e662-4299-8e12-b4b736b8b915-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-6l7tx\" (UID: \"4794d008-e662-4299-8e12-b4b736b8b915\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:33.004562 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.004456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679f5\" (UniqueName: \"kubernetes.io/projected/d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23-kube-api-access-679f5\") pod \"downloads-6594f4c75-xxgdj\" (UID: \"d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23\") " pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:33.006686 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.006668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4794d008-e662-4299-8e12-b4b736b8b915-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-6l7tx\" (UID: \"4794d008-e662-4299-8e12-b4b736b8b915\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:33.018028 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.017998    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-679f5\" (UniqueName: \"kubernetes.io/projected/d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23-kube-api-access-679f5\") pod \"downloads-6594f4c75-xxgdj\" (UID: \"d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23\") " pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:33.173959 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.173939    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:33.179805 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.179789    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:33.456885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.456804    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"]
Sep 17 15:57:33.459674 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:33.459647    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4794d008_e662_4299_8e12_b4b736b8b915.slice/crio-1d97fcb6f6f1cf82a4283f376a1e57dc6627ac44efdd643cd4bf589984922183 WatchSource:0}: Error finding container 1d97fcb6f6f1cf82a4283f376a1e57dc6627ac44efdd643cd4bf589984922183: Status 404 returned error can't find the container with id 1d97fcb6f6f1cf82a4283f376a1e57dc6627ac44efdd643cd4bf589984922183
Sep 17 15:57:33.462131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:33.462109    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-xxgdj"]
Sep 17 15:57:33.464828 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:33.464802    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2620d2a_0f4c_4c8d_a6af_6921ec7fbc23.slice/crio-0996c9b833064f7180c4a5ee13166c9bea1d3360becefa3859adbbddc17465b6 WatchSource:0}: Error finding container 0996c9b833064f7180c4a5ee13166c9bea1d3360becefa3859adbbddc17465b6: Status 404 returned error can't find the container with id 0996c9b833064f7180c4a5ee13166c9bea1d3360becefa3859adbbddc17465b6
Sep 17 15:57:34.411937 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:34.411865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx" event={"ID":"4794d008-e662-4299-8e12-b4b736b8b915","Type":"ContainerStarted","Data":"1d97fcb6f6f1cf82a4283f376a1e57dc6627ac44efdd643cd4bf589984922183"}
Sep 17 15:57:34.413401 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:34.413368    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-xxgdj" event={"ID":"d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23","Type":"ContainerStarted","Data":"0996c9b833064f7180c4a5ee13166c9bea1d3360becefa3859adbbddc17465b6"}
Sep 17 15:57:35.418846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.418805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx" event={"ID":"4794d008-e662-4299-8e12-b4b736b8b915","Type":"ContainerStarted","Data":"a8f3a6f012ddc3f12f17044630434515c6824abe85ac26e821d145966577021d"}
Sep 17 15:57:35.419282 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.419031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:35.424282 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.424254    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx"
Sep 17 15:57:35.433283 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.433234    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-6l7tx" podStartSLOduration=2.349929194 podStartE2EDuration="3.433220944s" totalImagesPullingTime="1.08329175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:35.431629248 +0000 UTC m=+130.730773742" watchObservedRunningTime="2026-09-17 15:57:35.433220944 +0000 UTC m=+130.732365422"
Sep 17 15:57:35.770794 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.770717    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"]
Sep 17 15:57:35.773519 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.773504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.776259 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.776031    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:57:35.776259 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.776086    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 15:57:35.776442 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.776299    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 15:57:35.777027 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.776996    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ns87k\""
Sep 17 15:57:35.777027 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.777020    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:57:35.777204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.777004    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:57:35.781107 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.781081    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"]
Sep 17 15:57:35.825311 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.825284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.825455 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.825364    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.825455 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.825392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.825455 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.825443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnx4t\" (UniqueName: \"kubernetes.io/projected/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-kube-api-access-gnx4t\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.926778 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.926744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.926915 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.926799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.926915 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:35.926882    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 15:57:35.926982 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:35.926936    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-tls podName:0558ae32-24f5-45d5-b55a-4e835c0c9b2d nodeName:}" failed. No retries permitted until 2026-09-17 15:57:36.426924691 +0000 UTC m=+131.726069148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-zsj2s" (UID: "0558ae32-24f5-45d5-b55a-4e835c0c9b2d") : secret "prometheus-operator-tls" not found
Sep 17 15:57:35.926982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.926923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.926982 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.926971    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnx4t\" (UniqueName: \"kubernetes.io/projected/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-kube-api-access-gnx4t\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.927486 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.927464    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.929171 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.929145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:35.935844 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:35.935822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnx4t\" (UniqueName: \"kubernetes.io/projected/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-kube-api-access-gnx4t\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:36.430082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.430049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:36.432805 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.432778    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0558ae32-24f5-45d5-b55a-4e835c0c9b2d-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-zsj2s\" (UID: \"0558ae32-24f5-45d5-b55a-4e835c0c9b2d\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:36.683111 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.683031    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"
Sep 17 15:57:36.737403 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.737239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:57:36.737403 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.737353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:57:36.740372 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.740223    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:57:36.741088 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.740958    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:57:36.751967 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.751942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e376dd58-f13e-43de-bb4f-e2cac00b537f-metrics-certs\") pod \"network-metrics-daemon-6kjfr\" (UID: \"e376dd58-f13e-43de-bb4f-e2cac00b537f\") " pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:57:36.752489 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.752460    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d18cce37-76b4-406d-b113-b47a3c850d05-original-pull-secret\") pod \"global-pull-secret-syncer-vqfpz\" (UID: \"d18cce37-76b4-406d-b113-b47a3c850d05\") " pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:57:36.838670 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.838637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:57:36.842577 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.842557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfpz"
Sep 17 15:57:36.843182 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.843158    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:57:36.843851 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.843833    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v87nd\""
Sep 17 15:57:36.848913 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.848891    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kjfr"
Sep 17 15:57:36.851699 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.851674    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:57:36.861911 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.861863    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ptw\" (UniqueName: \"kubernetes.io/projected/c161efdf-95e9-4b7a-9c4a-686c6af3e9a3-kube-api-access-v5ptw\") pod \"network-check-target-m9jbf\" (UID: \"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3\") " pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:57:36.964636 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:36.964305    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s"]
Sep 17 15:57:36.966936 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:36.966893    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0558ae32_24f5_45d5_b55a_4e835c0c9b2d.slice/crio-0d19809ce78eeb1495ce2e3aa11a0424eab785a08e53aa0c8b3229ddebab0645 WatchSource:0}: Error finding container 0d19809ce78eeb1495ce2e3aa11a0424eab785a08e53aa0c8b3229ddebab0645: Status 404 returned error can't find the container with id 0d19809ce78eeb1495ce2e3aa11a0424eab785a08e53aa0c8b3229ddebab0645
Sep 17 15:57:37.122295 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.122261    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqfpz"]
Sep 17 15:57:37.125255 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:37.125212    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18cce37_76b4_406d_b113_b47a3c850d05.slice/crio-374aff56e25e053d50316aee43f273fc81ae21d0ee51e4bf1e7b77b50fbd9b7a WatchSource:0}: Error finding container 374aff56e25e053d50316aee43f273fc81ae21d0ee51e4bf1e7b77b50fbd9b7a: Status 404 returned error can't find the container with id 374aff56e25e053d50316aee43f273fc81ae21d0ee51e4bf1e7b77b50fbd9b7a
Sep 17 15:57:37.135629 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.135611    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fvxq9\""
Sep 17 15:57:37.138358 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.138337    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kjfr"]
Sep 17 15:57:37.141791 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:37.141767    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode376dd58_f13e_43de_bb4f_e2cac00b537f.slice/crio-6f5b269d110f88b3ae4437e3bf0280da697b4a0baa66aded1784bbe77196c60d WatchSource:0}: Error finding container 6f5b269d110f88b3ae4437e3bf0280da697b4a0baa66aded1784bbe77196c60d: Status 404 returned error can't find the container with id 6f5b269d110f88b3ae4437e3bf0280da697b4a0baa66aded1784bbe77196c60d
Sep 17 15:57:37.143177 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.143151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:57:37.392888 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.392807    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c6q8r"
Sep 17 15:57:37.422469 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.422439    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9jbf"]
Sep 17 15:57:37.426543 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:37.426493    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc161efdf_95e9_4b7a_9c4a_686c6af3e9a3.slice/crio-ac86540065c115b49c1f0b8e288a64807d495b10e2ed33ad9da2409c00403a04 WatchSource:0}: Error finding container ac86540065c115b49c1f0b8e288a64807d495b10e2ed33ad9da2409c00403a04: Status 404 returned error can't find the container with id ac86540065c115b49c1f0b8e288a64807d495b10e2ed33ad9da2409c00403a04
Sep 17 15:57:37.429660 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.429631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqfpz" event={"ID":"d18cce37-76b4-406d-b113-b47a3c850d05","Type":"ContainerStarted","Data":"374aff56e25e053d50316aee43f273fc81ae21d0ee51e4bf1e7b77b50fbd9b7a"}
Sep 17 15:57:37.431482 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.431429    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kjfr" event={"ID":"e376dd58-f13e-43de-bb4f-e2cac00b537f","Type":"ContainerStarted","Data":"6f5b269d110f88b3ae4437e3bf0280da697b4a0baa66aded1784bbe77196c60d"}
Sep 17 15:57:37.433240 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:37.433215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s" event={"ID":"0558ae32-24f5-45d5-b55a-4e835c0c9b2d","Type":"ContainerStarted","Data":"0d19809ce78eeb1495ce2e3aa11a0424eab785a08e53aa0c8b3229ddebab0645"}
Sep 17 15:57:38.439217 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:38.439176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9jbf" event={"ID":"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3","Type":"ContainerStarted","Data":"ac86540065c115b49c1f0b8e288a64807d495b10e2ed33ad9da2409c00403a04"}
Sep 17 15:57:39.448116 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:39.447556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kjfr" event={"ID":"e376dd58-f13e-43de-bb4f-e2cac00b537f","Type":"ContainerStarted","Data":"fe172f4a94ac8e3fcb25aecccdd0e43fe7bf93e31e8453bb792d85159b4eda90"}
Sep 17 15:57:39.448116 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:39.447672    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kjfr" event={"ID":"e376dd58-f13e-43de-bb4f-e2cac00b537f","Type":"ContainerStarted","Data":"8bb27a33b3264ccfd114c2910fb5304fa5d938076a2b1f324216176862753aae"}
Sep 17 15:57:39.450624 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:39.450389    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s" event={"ID":"0558ae32-24f5-45d5-b55a-4e835c0c9b2d","Type":"ContainerStarted","Data":"39b22c38c96a2acc33b4d985995e44e3c5b92503224672143cf969787ab80403"}
Sep 17 15:57:39.450624 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:39.450440    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s" event={"ID":"0558ae32-24f5-45d5-b55a-4e835c0c9b2d","Type":"ContainerStarted","Data":"9162f972c3a0d611bcecc5ede2bc48fab3879b3a99017c847e8f8cc8d5ef9fa4"}
Sep 17 15:57:39.464678 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:39.463935    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6kjfr" podStartSLOduration=130.744401561 podStartE2EDuration="2m12.46391825s" totalImagesPullingTime="1.719516689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:39.463917393 +0000 UTC m=+134.763061892" watchObservedRunningTime="2026-09-17 15:57:39.46391825 +0000 UTC m=+134.763062729"
Sep 17 15:57:39.479733 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:39.479253    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-zsj2s" podStartSLOduration=2.583743644 podStartE2EDuration="4.479233498s" totalImagesPullingTime="1.895489854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:39.478616444 +0000 UTC m=+134.777760941" watchObservedRunningTime="2026-09-17 15:57:39.479233498 +0000 UTC m=+134.778377972"
Sep 17 15:57:41.176717 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.176455    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"]
Sep 17 15:57:41.180099 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.180072    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.183724 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.183547    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:57:41.183724 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.183553    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 15:57:41.184195 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.184176    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gj6m5\""
Sep 17 15:57:41.191685 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.191610    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"]
Sep 17 15:57:41.201469 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.201444    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"]
Sep 17 15:57:41.205059 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.205038    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.208888 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.208862    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 15:57:41.209270 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.209121    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:57:41.209270 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.209153    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vp9sm\""
Sep 17 15:57:41.209885 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.209722    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 15:57:41.216782 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.216233    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"]
Sep 17 15:57:41.220907 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.220886    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tvzxb"]
Sep 17 15:57:41.224434 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.224411    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.227877 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.227748    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t8vk7\""
Sep 17 15:57:41.227877 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.227833    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:57:41.228041 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.227950    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:57:41.228101 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.228073    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:57:41.281978 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.281935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc2a21ad-52bd-4169-abde-2d74fbd473ee-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.281982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282037    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67qw\" (UniqueName: \"kubernetes.io/projected/fc2a21ad-52bd-4169-abde-2d74fbd473ee-kube-api-access-x67qw\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4x4\" (UniqueName: \"kubernetes.io/projected/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-kube-api-access-lm4x4\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-wtmp\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282114    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-textfile\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.282151 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt298\" (UniqueName: \"kubernetes.io/projected/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-api-access-zt298\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-metrics-client-ca\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-tls\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-sys\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282317    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-root\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282358    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.282627 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.282383    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.383650 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383609    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc2a21ad-52bd-4169-abde-2d74fbd473ee-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383707    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67qw\" (UniqueName: \"kubernetes.io/projected/fc2a21ad-52bd-4169-abde-2d74fbd473ee-kube-api-access-x67qw\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383760    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383788    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4x4\" (UniqueName: \"kubernetes.io/projected/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-kube-api-access-lm4x4\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:41.383791    2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-wtmp\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.383837 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-textfile\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:41.383874    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-tls podName:fc2a21ad-52bd-4169-abde-2d74fbd473ee nodeName:}" failed. No retries permitted until 2026-09-17 15:57:41.883852309 +0000 UTC m=+137.182996770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-2glt4" (UID: "fc2a21ad-52bd-4169-abde-2d74fbd473ee") : secret "openshift-state-metrics-tls" not found
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt298\" (UniqueName: \"kubernetes.io/projected/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-api-access-zt298\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.383963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-metrics-client-ca\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-tls\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-sys\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384095    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384148    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-textfile\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384261 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384154    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-root\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384859 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384449    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-sys\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.384859 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc2a21ad-52bd-4169-abde-2d74fbd473ee-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.384859 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.385022 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384856    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.385022 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.384990    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-metrics-client-ca\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.385139 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.385059    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.385447 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:41.385423    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:57:41.385544 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:41.385480    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-tls podName:e9799fab-3ab4-43fd-a620-4dabcb0f9a6c nodeName:}" failed. No retries permitted until 2026-09-17 15:57:41.88546446 +0000 UTC m=+137.184608931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-tls") pod "node-exporter-tvzxb" (UID: "e9799fab-3ab4-43fd-a620-4dabcb0f9a6c") : secret "node-exporter-tls" not found
Sep 17 15:57:41.385544 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.385527    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.385692 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.385610    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-root\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.385692 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.385622    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-wtmp\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.387455 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.387404    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.387871 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.387849    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.388180 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.388137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.389237 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.389198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.404342 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.404311    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4x4\" (UniqueName: \"kubernetes.io/projected/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-kube-api-access-lm4x4\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.405188 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.405143    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt298\" (UniqueName: \"kubernetes.io/projected/8de00618-a4d1-4ec2-9a15-b634ffb3cc89-kube-api-access-zt298\") pod \"kube-state-metrics-84c885658c-g2xtr\" (UID: \"8de00618-a4d1-4ec2-9a15-b634ffb3cc89\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.406139 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.406115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67qw\" (UniqueName: \"kubernetes.io/projected/fc2a21ad-52bd-4169-abde-2d74fbd473ee-kube-api-access-x67qw\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.517787 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.517707    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"
Sep 17 15:57:41.889422 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.888933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:41.889422 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.889026    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-tls\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.891981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.891952    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9799fab-3ab4-43fd-a620-4dabcb0f9a6c-node-exporter-tls\") pod \"node-exporter-tvzxb\" (UID: \"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c\") " pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:41.892111 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:41.892014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc2a21ad-52bd-4169-abde-2d74fbd473ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2glt4\" (UID: \"fc2a21ad-52bd-4169-abde-2d74fbd473ee\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:42.093074 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.093035    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"
Sep 17 15:57:42.137416 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.137379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvzxb"
Sep 17 15:57:42.260275 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.260115    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:42.264464 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.264442    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.268439 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268275    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:57:42.268439 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268296    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:57:42.268439 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268346    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:57:42.268439 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268295    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:57:42.268744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268562    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:57:42.268744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268636    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:57:42.268744 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268675    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:57:42.268903 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268776    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:57:42.268903 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268814    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xxbbp\""
Sep 17 15:57:42.268903 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.268777    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:57:42.275057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.274905    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:42.392872 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.392872 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.392872 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-out\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.392990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.393016    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.393051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94dcf\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-kube-api-access-94dcf\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.393078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393131 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.393108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393514 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.393138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.393514 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.393174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-web-config\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493568    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-out\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493669    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493785    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94dcf\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-kube-api-access-94dcf\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.493921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.493844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.494653    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.494653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.494740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.494779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-web-config\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.494812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.494878    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.495204 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:57:42.494910    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle podName:f9f02859-1691-4cd3-b67f-3b06f8e666cc nodeName:}" failed. No retries permitted until 2026-09-17 15:57:42.994885889 +0000 UTC m=+138.294030357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc") : configmap references non-existent config key: ca-bundle.crt
Sep 17 15:57:42.498316 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.498006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.498316 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.498070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.498875 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.498850    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.499538 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.499080    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-out\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.499538 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.499387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-web-config\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.499538 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.499500    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.500286 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.500232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.500519 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.500471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.505933 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.505902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.508576 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.508523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94dcf\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-kube-api-access-94dcf\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.784112 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:42.783743    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9799fab_3ab4_43fd_a620_4dabcb0f9a6c.slice/crio-b56eca564640540da82329607012dade285e04c20825658765b7e1da8a3e8c96 WatchSource:0}: Error finding container b56eca564640540da82329607012dade285e04c20825658765b7e1da8a3e8c96: Status 404 returned error can't find the container with id b56eca564640540da82329607012dade285e04c20825658765b7e1da8a3e8c96
Sep 17 15:57:42.893114 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.893051    2557 patch_prober.go:28] interesting pod/image-registry-69c8fb8969-jdnhs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:57:42.893255 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.893115    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" podUID="017d8d03-3f22-469b-bebe-58c6258404ac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:57:42.998973 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.998934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:42.999922 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:42.999895    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:43.060332 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.060296    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-g2xtr"]
Sep 17 15:57:43.063527 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:43.063492    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de00618_a4d1_4ec2_9a15_b634ffb3cc89.slice/crio-b666d9ee5c4ea57bde1df5c9d700295a2fca6aa86cd58e4bc38f1a7966e70199 WatchSource:0}: Error finding container b666d9ee5c4ea57bde1df5c9d700295a2fca6aa86cd58e4bc38f1a7966e70199: Status 404 returned error can't find the container with id b666d9ee5c4ea57bde1df5c9d700295a2fca6aa86cd58e4bc38f1a7966e70199
Sep 17 15:57:43.064484 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.064458    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4"]
Sep 17 15:57:43.066834 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:43.066720    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2a21ad_52bd_4169_abde_2d74fbd473ee.slice/crio-8fccfddb3ec18ddd2a93c5ad8dde19f21b9a14de804310df3fd4e0347faa605c WatchSource:0}: Error finding container 8fccfddb3ec18ddd2a93c5ad8dde19f21b9a14de804310df3fd4e0347faa605c: Status 404 returned error can't find the container with id 8fccfddb3ec18ddd2a93c5ad8dde19f21b9a14de804310df3fd4e0347faa605c
Sep 17 15:57:43.183398 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.183364    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:43.471037 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.470988    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqfpz" event={"ID":"d18cce37-76b4-406d-b113-b47a3c850d05","Type":"ContainerStarted","Data":"8c2e97d411aa619772c0e6d5670380a103f70a53ea5192bbb7ce7986e407a277"}
Sep 17 15:57:43.472637 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.472608    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr" event={"ID":"8de00618-a4d1-4ec2-9a15-b634ffb3cc89","Type":"ContainerStarted","Data":"b666d9ee5c4ea57bde1df5c9d700295a2fca6aa86cd58e4bc38f1a7966e70199"}
Sep 17 15:57:43.475271 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.475242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4" event={"ID":"fc2a21ad-52bd-4169-abde-2d74fbd473ee","Type":"ContainerStarted","Data":"95e7ed06d2f17cb954cef0f08d1d73becafa16cc60bc8f70671607ec35e56fce"}
Sep 17 15:57:43.475629 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.475514    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4" event={"ID":"fc2a21ad-52bd-4169-abde-2d74fbd473ee","Type":"ContainerStarted","Data":"035a78f2b3954e0d8f921d9c51d0a0f260a02542f4a4e59bde62dc88a2d6f779"}
Sep 17 15:57:43.475629 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.475574    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4" event={"ID":"fc2a21ad-52bd-4169-abde-2d74fbd473ee","Type":"ContainerStarted","Data":"8fccfddb3ec18ddd2a93c5ad8dde19f21b9a14de804310df3fd4e0347faa605c"}
Sep 17 15:57:43.478231 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.478205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9jbf" event={"ID":"c161efdf-95e9-4b7a-9c4a-686c6af3e9a3","Type":"ContainerStarted","Data":"ed47374bf40a11535ac2e25882476677bceeb0e5a75b7e4e8dd118ec7a7f7b4a"}
Sep 17 15:57:43.478520 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.478498    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:43.478657 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.478643    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:57:43.480218 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.480195    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvzxb" event={"ID":"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c","Type":"ContainerStarted","Data":"b56eca564640540da82329607012dade285e04c20825658765b7e1da8a3e8c96"}
Sep 17 15:57:43.481347 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:43.481319    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f02859_1691_4cd3_b67f_3b06f8e666cc.slice/crio-3917ba5873b58f5c1f13abecb5ecbe58fd88fa073d719993c519a72763f27ebd WatchSource:0}: Error finding container 3917ba5873b58f5c1f13abecb5ecbe58fd88fa073d719993c519a72763f27ebd: Status 404 returned error can't find the container with id 3917ba5873b58f5c1f13abecb5ecbe58fd88fa073d719993c519a72763f27ebd
Sep 17 15:57:43.490574 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.490521    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vqfpz" podStartSLOduration=130.821645709 podStartE2EDuration="2m16.490503143s" totalImagesPullingTime="5.668857434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:43.489004844 +0000 UTC m=+138.788149325" watchObservedRunningTime="2026-09-17 15:57:43.490503143 +0000 UTC m=+138.789647623"
Sep 17 15:57:43.514022 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:43.513965    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m9jbf" podStartSLOduration=131.163573115 podStartE2EDuration="2m16.513948942s" totalImagesPullingTime="5.350375827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:43.512919613 +0000 UTC m=+138.812064113" watchObservedRunningTime="2026-09-17 15:57:43.513948942 +0000 UTC m=+138.813093419"
Sep 17 15:57:44.487438 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:44.487400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"3917ba5873b58f5c1f13abecb5ecbe58fd88fa073d719993c519a72763f27ebd"}
Sep 17 15:57:44.490136 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:44.490087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvzxb" event={"ID":"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c","Type":"ContainerStarted","Data":"f75394bdb82819a36bffe7b9c379f54e323f109d0cc616c21ac76677180dc3f1"}
Sep 17 15:57:45.497141 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.497077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4" event={"ID":"fc2a21ad-52bd-4169-abde-2d74fbd473ee","Type":"ContainerStarted","Data":"258dd79a1f3e6bfb1a2fc36b1873dce1603bf264e77d2710051654390cec1d29"}
Sep 17 15:57:45.499451 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.499272    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"}
Sep 17 15:57:45.501759 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.501731    2557 generic.go:356] "Generic (PLEG): container finished" podID="e9799fab-3ab4-43fd-a620-4dabcb0f9a6c" containerID="f75394bdb82819a36bffe7b9c379f54e323f109d0cc616c21ac76677180dc3f1" exitCode=0
Sep 17 15:57:45.501861 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.501818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvzxb" event={"ID":"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c","Type":"ContainerDied","Data":"f75394bdb82819a36bffe7b9c379f54e323f109d0cc616c21ac76677180dc3f1"}
Sep 17 15:57:45.501923 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.501865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvzxb" event={"ID":"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c","Type":"ContainerStarted","Data":"091789133cc1cb3b44a8ad8820c0371f9f6e25d41dc9d5a84956703654013eac"}
Sep 17 15:57:45.503813 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.503781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr" event={"ID":"8de00618-a4d1-4ec2-9a15-b634ffb3cc89","Type":"ContainerStarted","Data":"049780b0d03e0cf515b0f5983fd38b7e78266463ab3c2c201c5894aadc0af5e7"}
Sep 17 15:57:45.518691 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:45.518641    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2glt4" podStartSLOduration=2.624987761 podStartE2EDuration="4.518623164s" totalImagesPullingTime="1.893635403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:45.517185633 +0000 UTC m=+140.816330158" watchObservedRunningTime="2026-09-17 15:57:45.518623164 +0000 UTC m=+140.817767643"
Sep 17 15:57:46.423336 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.423303    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f658797d9-cpzvn"]
Sep 17 15:57:46.427038 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.427013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.431859 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.431833    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 15:57:46.431991 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.431833    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 15:57:46.432581 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.432559    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 15:57:46.432843 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.432825    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 15:57:46.432912 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.432825    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-254v5\""
Sep 17 15:57:46.433159 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.433139    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 15:57:46.441392 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.441369    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 15:57:46.446543 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.446517    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f658797d9-cpzvn"]
Sep 17 15:57:46.530608 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-telemeter-client-tls\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-metrics-client-ca\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530683    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbgq\" (UniqueName: \"kubernetes.io/projected/0e64ec39-be7d-4675-aa5b-bd6ff8244541-kube-api-access-gwbgq\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-serving-certs-ca-bundle\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530815    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-secret-telemeter-client\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.531057 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.530878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-federate-client-tls\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.631954 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.631906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-secret-telemeter-client\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632117 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-federate-client-tls\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632203 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632152    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-telemeter-client-tls\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632481 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-metrics-client-ca\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632634 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbgq\" (UniqueName: \"kubernetes.io/projected/0e64ec39-be7d-4675-aa5b-bd6ff8244541-kube-api-access-gwbgq\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632702 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-serving-certs-ca-bundle\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632755 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.632822 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.632751    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.633904 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.633668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.633904 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.633852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-serving-certs-ca-bundle\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.634070 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.634000    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e64ec39-be7d-4675-aa5b-bd6ff8244541-metrics-client-ca\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.635524 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.635498    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-telemeter-client-tls\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.635661 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.635572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-secret-telemeter-client\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.635725 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.635685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-federate-client-tls\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.635980 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.635958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e64ec39-be7d-4675-aa5b-bd6ff8244541-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.642968 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.642946    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbgq\" (UniqueName: \"kubernetes.io/projected/0e64ec39-be7d-4675-aa5b-bd6ff8244541-kube-api-access-gwbgq\") pod \"telemeter-client-f658797d9-cpzvn\" (UID: \"0e64ec39-be7d-4675-aa5b-bd6ff8244541\") " pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:46.737244 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:46.737167    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn"
Sep 17 15:57:47.518551 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.518477    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:47.523501 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.523291    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.534888 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.534861    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:57:47.535235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.534923    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:57:47.535235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.534872    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:57:47.535235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.535140    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:57:47.535235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.535145    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:57:47.543905 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.543828    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.547560    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.547640    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.547808    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.547563    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9fi2oo75t6ujn\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.548001    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.548035    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.548145    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:57:47.548716 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.548187    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:57:47.551450 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.551431    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:57:47.554758 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.554735    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6g7qb\""
Sep 17 15:57:47.564494 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.564468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:47.565513 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.565401    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:57:47.640030 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.639995    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640215 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640040    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640215 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640215 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640111    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640274    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640346 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640532 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640532 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640532 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640680 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-web-config\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640680 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640587    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640680 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640794 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640690    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640794 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640711    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85c6\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-kube-api-access-z85c6\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640794 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640934 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640934 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config-out\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.640934 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.640863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742038 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.741991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742038 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742186    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-web-config\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742277 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742270    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742321    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85c6\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-kube-api-access-z85c6\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742408    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config-out\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742516    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742539    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.742745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742578    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.743223 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.742879    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.743223 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.743067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.743223 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.743093    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.744676 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.743534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.744676 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.744233    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.746365 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.746343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.746543 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.746512    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.747029 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.747003    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.747029 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.747023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.748139 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.748120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.748368 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.748341    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.749382 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.749331    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-web-config\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.749851 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.749807    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.750856 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.750806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.750856 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.750827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.751076 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.751053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.751396 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.751374    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.752993 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.752953    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.753416 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.753396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config-out\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.759177 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.759146    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85c6\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-kube-api-access-z85c6\") pod \"prometheus-k8s-0\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:47.836616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:47.836512    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:52.539244 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:52.538103    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78" exitCode=0
Sep 17 15:57:52.539244 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:52.538167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"}
Sep 17 15:57:52.747434 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:52.747392    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f658797d9-cpzvn"]
Sep 17 15:57:52.750353 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:52.750070    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:52.753565 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:57:52.753169    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c73b93_2548_4d1d_a158_1d9a7a0fefae.slice/crio-4fa1101a3731d2299a8865350774b9e06a42fc055e0dab7c712bf81632322df7 WatchSource:0}: Error finding container 4fa1101a3731d2299a8865350774b9e06a42fc055e0dab7c712bf81632322df7: Status 404 returned error can't find the container with id 4fa1101a3731d2299a8865350774b9e06a42fc055e0dab7c712bf81632322df7
Sep 17 15:57:52.889073 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:52.889048    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:53.550123 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.550073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-xxgdj" event={"ID":"d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23","Type":"ContainerStarted","Data":"27d3dc19a6d2d7ee10f2ce4da50834625179cf88eb1d95980223c2976695de1c"}
Sep 17 15:57:53.551220 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.551192    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:53.556983 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.556936    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-xxgdj"
Sep 17 15:57:53.558554 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.557908    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvzxb" event={"ID":"e9799fab-3ab4-43fd-a620-4dabcb0f9a6c","Type":"ContainerStarted","Data":"7f591339e29db129e1b6b45fb835c2b7085477080a1c00d1a2132d3a539cb4a7"}
Sep 17 15:57:53.563362 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.563282    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn" event={"ID":"0e64ec39-be7d-4675-aa5b-bd6ff8244541","Type":"ContainerStarted","Data":"5847a7ab95d3d44915952f53a433bd36c0b3fdb2d3ed8e777c4c9b20f7444c65"}
Sep 17 15:57:53.569177 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.568953    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33" exitCode=0
Sep 17 15:57:53.569177 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.569047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"}
Sep 17 15:57:53.569177 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.569105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"4fa1101a3731d2299a8865350774b9e06a42fc055e0dab7c712bf81632322df7"}
Sep 17 15:57:53.571615 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.571548    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-xxgdj" podStartSLOduration=2.479040739 podStartE2EDuration="21.571529419s" totalImagesPullingTime="19.09248868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:53.569003584 +0000 UTC m=+148.868148065" watchObservedRunningTime="2026-09-17 15:57:53.571529419 +0000 UTC m=+148.870673896"
Sep 17 15:57:53.607898 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.600770    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr" event={"ID":"8de00618-a4d1-4ec2-9a15-b634ffb3cc89","Type":"ContainerStarted","Data":"10f853704a40335a4d633295a845a3326c3849f8560e214b09db3d3a890c793c"}
Sep 17 15:57:53.607898 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.601133    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr" event={"ID":"8de00618-a4d1-4ec2-9a15-b634ffb3cc89","Type":"ContainerStarted","Data":"ce73f784fc16a61f7abf4d036118f587236450bafd4aad9be34f7f0cffbe9d4f"}
Sep 17 15:57:53.611709 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.610652    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tvzxb" podStartSLOduration=11.473075053 podStartE2EDuration="12.610634758s" totalImagesPullingTime="1.057266129s" totalInitContainerRuntime="80.293576ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:53.610038058 +0000 UTC m=+148.909182561" watchObservedRunningTime="2026-09-17 15:57:53.610634758 +0000 UTC m=+148.909779235"
Sep 17 15:57:53.668712 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:53.667398    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-g2xtr" podStartSLOduration=10.621058456 podStartE2EDuration="12.667376093s" totalImagesPullingTime="2.046317637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:53.636569637 +0000 UTC m=+148.935714327" watchObservedRunningTime="2026-09-17 15:57:53.667376093 +0000 UTC m=+148.966520573"
Sep 17 15:57:56.621258 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.621172    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn" event={"ID":"0e64ec39-be7d-4675-aa5b-bd6ff8244541","Type":"ContainerStarted","Data":"7edbf6380584c43851cd3b630a1425462fca1a22d30249c1c98bfa54b1ee01a1"}
Sep 17 15:57:56.621820 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.621260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn" event={"ID":"0e64ec39-be7d-4675-aa5b-bd6ff8244541","Type":"ContainerStarted","Data":"2bd860435a8dfbe7c85de5a168480677b570b97c9d51c02f987c79311804dd8b"}
Sep 17 15:57:56.621820 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.621361    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn" event={"ID":"0e64ec39-be7d-4675-aa5b-bd6ff8244541","Type":"ContainerStarted","Data":"d2152f4cfc2f7b45ea3330a58d374ab3a7d97bcc75d5e3683b7e3a1d80ce57d1"}
Sep 17 15:57:56.625705 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.625674    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"}
Sep 17 15:57:56.625924 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.625787    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"}
Sep 17 15:57:56.625924 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.625870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"}
Sep 17 15:57:56.625924 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.625918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"}
Sep 17 15:57:56.626090 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.625969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"}
Sep 17 15:57:56.647647 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:56.647567    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f658797d9-cpzvn" podStartSLOduration=7.725404572 podStartE2EDuration="10.647548881s" totalImagesPullingTime="2.922144309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:56.644030803 +0000 UTC m=+151.943175299" watchObservedRunningTime="2026-09-17 15:57:56.647548881 +0000 UTC m=+151.946693362"
Sep 17 15:57:57.901705 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:57.901615    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" podUID="017d8d03-3f22-469b-bebe-58c6258404ac" containerName="registry" containerID="cri-o://9e500b752b9c3d5dfc994c81fab132c637ee087b043724c00bbb0c0b277b1a65" gracePeriod=30
Sep 17 15:57:58.638473 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:58.638382    2557 generic.go:356] "Generic (PLEG): container finished" podID="017d8d03-3f22-469b-bebe-58c6258404ac" containerID="9e500b752b9c3d5dfc994c81fab132c637ee087b043724c00bbb0c0b277b1a65" exitCode=0
Sep 17 15:57:58.638651 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:58.638466    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" event={"ID":"017d8d03-3f22-469b-bebe-58c6258404ac","Type":"ContainerDied","Data":"9e500b752b9c3d5dfc994c81fab132c637ee087b043724c00bbb0c0b277b1a65"}
Sep 17 15:57:59.054367 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.054339    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:59.154820 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.154793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hlwp\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-kube-api-access-8hlwp\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.154928 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.154860    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-registry-certificates\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.154928 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.154899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/017d8d03-3f22-469b-bebe-58c6258404ac-ca-trust-extracted\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.154928 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.154922    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.155082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.154956    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-image-registry-private-configuration\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.155082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.155011    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-installation-pull-secrets\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.155082 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.155039    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-bound-sa-token\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.155235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.155097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-trusted-ca\") pod \"017d8d03-3f22-469b-bebe-58c6258404ac\" (UID: \"017d8d03-3f22-469b-bebe-58c6258404ac\") "
Sep 17 15:57:59.155235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.155218    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:59.155375 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.155348    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-registry-certificates\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.156343 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.156274    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:59.158205 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.158172    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:59.158352 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.158326    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:59.158761 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.158686    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:59.158901 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.158831    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:59.160670 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.160643    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-kube-api-access-8hlwp" (OuterVolumeSpecName: "kube-api-access-8hlwp") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "kube-api-access-8hlwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:59.167358 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.167334    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d8d03-3f22-469b-bebe-58c6258404ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "017d8d03-3f22-469b-bebe-58c6258404ac" (UID: "017d8d03-3f22-469b-bebe-58c6258404ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:59.256318 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256283    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlwp\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-kube-api-access-8hlwp\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.256318 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256318    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/017d8d03-3f22-469b-bebe-58c6258404ac-ca-trust-extracted\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.256549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256330    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-registry-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.256549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256343    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-image-registry-private-configuration\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.256549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256355    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/017d8d03-3f22-469b-bebe-58c6258404ac-installation-pull-secrets\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.256549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256367    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/017d8d03-3f22-469b-bebe-58c6258404ac-bound-sa-token\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.256549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.256378    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/017d8d03-3f22-469b-bebe-58c6258404ac-trusted-ca\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:57:59.644841 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.644761    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs"
Sep 17 15:57:59.644841 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.644777    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8fb8969-jdnhs" event={"ID":"017d8d03-3f22-469b-bebe-58c6258404ac","Type":"ContainerDied","Data":"d4e81c125a6c689b0117da83ab446e71dbd697bb2324872300ec5b52cca26f8f"}
Sep 17 15:57:59.645047 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.644843    2557 scope.go:117] "RemoveContainer" containerID="9e500b752b9c3d5dfc994c81fab132c637ee087b043724c00bbb0c0b277b1a65"
Sep 17 15:57:59.657870 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.657803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerStarted","Data":"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"}
Sep 17 15:57:59.661402 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.661379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"}
Sep 17 15:57:59.661502 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.661445    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"}
Sep 17 15:57:59.686249 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.686184    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.106501745 podStartE2EDuration="17.686165157s" totalImagesPullingTime="7.557669436s" totalInitContainerRuntime="21.993976ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:59.684299211 +0000 UTC m=+154.983443738" watchObservedRunningTime="2026-09-17 15:57:59.686165157 +0000 UTC m=+154.985309639"
Sep 17 15:57:59.701454 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.701422    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8fb8969-jdnhs"]
Sep 17 15:57:59.708274 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:57:59.708244    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69c8fb8969-jdnhs"]
Sep 17 15:58:01.028929 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:01.028888    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017d8d03-3f22-469b-bebe-58c6258404ac" path="/var/lib/kubelet/pods/017d8d03-3f22-469b-bebe-58c6258404ac/volumes"
Sep 17 15:58:02.681989 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:02.681951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"}
Sep 17 15:58:02.682463 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:02.682028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"}
Sep 17 15:58:02.682463 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:02.682069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"}
Sep 17 15:58:02.682463 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:02.682115    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerStarted","Data":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"}
Sep 17 15:58:02.717199 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:02.713012    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.7203361489999995 podStartE2EDuration="15.712990469s" totalImagesPullingTime="7.968718329s" totalInitContainerRuntime="23.935991ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:02.7080683 +0000 UTC m=+158.007212832" watchObservedRunningTime="2026-09-17 15:58:02.712990469 +0000 UTC m=+158.012134950"
Sep 17 15:58:02.837492 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:02.837463    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:14.493724 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:14.493607    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m9jbf"
Sep 17 15:58:47.837549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:47.837506    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:47.857122 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:47.857095    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:47.873263 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:58:47.873237    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:01.939677 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.939648    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:59:01.940154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.939909    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="alertmanager" containerID="cri-o://97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541" gracePeriod=120
Sep 17 15:59:01.940154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.939935    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy" containerID="cri-o://6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2" gracePeriod=120
Sep 17 15:59:01.940154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.939988    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="prom-label-proxy" containerID="cri-o://07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db" gracePeriod=120
Sep 17 15:59:01.940154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.940015    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="config-reloader" containerID="cri-o://2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150" gracePeriod=120
Sep 17 15:59:01.940154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.940027    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-metric" containerID="cri-o://45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829" gracePeriod=120
Sep 17 15:59:01.940154 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:01.939959    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-web" containerID="cri-o://24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9" gracePeriod=120
Sep 17 15:59:02.917726 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917689    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db" exitCode=0
Sep 17 15:59:02.917726 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917715    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2" exitCode=0
Sep 17 15:59:02.917726 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917721    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150" exitCode=0
Sep 17 15:59:02.917726 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917727    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541" exitCode=0
Sep 17 15:59:02.917981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917759    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"}
Sep 17 15:59:02.917981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917823    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"}
Sep 17 15:59:02.917981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"}
Sep 17 15:59:02.917981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:02.917875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"}
Sep 17 15:59:03.186019 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.185997    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:03.285123 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285089    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-web-config\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285133    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285160    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-cluster-tls-config\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285184    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-metrics-client-ca\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285216    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-tls-assets\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285262    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-out\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285304 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285289    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285345    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-volume\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285373    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94dcf\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-kube-api-access-94dcf\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285405    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-main-tls\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285440    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-main-db\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.285616 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285559    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:03.285873 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285708    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-metrics-client-ca\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.285928 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.285872    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:03.286428 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.286364    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:03.288491 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.288452    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.288918 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.288872    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-volume" (OuterVolumeSpecName: "config-volume") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.289013 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.288925    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:03.289083 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.289020    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.289083 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.289070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-kube-api-access-94dcf" (OuterVolumeSpecName: "kube-api-access-94dcf") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "kube-api-access-94dcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:03.289471 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.289447    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-out" (OuterVolumeSpecName: "config-out") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:03.292750 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.292587    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.299250 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.299226    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-web-config" (OuterVolumeSpecName: "web-config") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.386778 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386752    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.386935 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386786    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-web\") pod \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\" (UID: \"f9f02859-1691-4cd3-b67f-3b06f8e666cc\") "
Sep 17 15:59:03.386994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386944    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-out\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.386994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386956    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.386994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386968    2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-config-volume\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.386994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386976    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94dcf\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-kube-api-access-94dcf\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.386994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386984    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-main-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.386994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.386993    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9f02859-1691-4cd3-b67f-3b06f8e666cc-alertmanager-main-db\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.387221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.387001    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-web-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.387221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.387009    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.387221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.387017    2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-cluster-tls-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.387221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.387025    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9f02859-1691-4cd3-b67f-3b06f8e666cc-tls-assets\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.388917 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.388883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.388994 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.388949    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f9f02859-1691-4cd3-b67f-3b06f8e666cc" (UID: "f9f02859-1691-4cd3-b67f-3b06f8e666cc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:03.488391 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.488318    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.488391 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.488348    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9f02859-1691-4cd3-b67f-3b06f8e666cc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:03.925692 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925655    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829" exitCode=0
Sep 17 15:59:03.925692 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925686    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerID="24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9" exitCode=0
Sep 17 15:59:03.925936 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925732    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:03.925936 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925742    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"}
Sep 17 15:59:03.925936 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"}
Sep 17 15:59:03.925936 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9f02859-1691-4cd3-b67f-3b06f8e666cc","Type":"ContainerDied","Data":"3917ba5873b58f5c1f13abecb5ecbe58fd88fa073d719993c519a72763f27ebd"}
Sep 17 15:59:03.925936 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.925893    2557 scope.go:117] "RemoveContainer" containerID="07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"
Sep 17 15:59:03.934041 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.934021    2557 scope.go:117] "RemoveContainer" containerID="45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"
Sep 17 15:59:03.941139 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.941121    2557 scope.go:117] "RemoveContainer" containerID="6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"
Sep 17 15:59:03.947958 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.947944    2557 scope.go:117] "RemoveContainer" containerID="24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"
Sep 17 15:59:03.954653 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.954624    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:59:03.958418 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.958388    2557 scope.go:117] "RemoveContainer" containerID="2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"
Sep 17 15:59:03.961727 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.961707    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:59:03.965839 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.965822    2557 scope.go:117] "RemoveContainer" containerID="97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"
Sep 17 15:59:03.972447 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.972430    2557 scope.go:117] "RemoveContainer" containerID="0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"
Sep 17 15:59:03.978726 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.978708    2557 scope.go:117] "RemoveContainer" containerID="07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"
Sep 17 15:59:03.978977 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.978961    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db\": container with ID starting with 07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db not found: ID does not exist" containerID="07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"
Sep 17 15:59:03.979027 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.978984    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"} err="failed to get container status \"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db\": rpc error: code = NotFound desc = could not find container \"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db\": container with ID starting with 07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db not found: ID does not exist"
Sep 17 15:59:03.979027 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979012    2557 scope.go:117] "RemoveContainer" containerID="45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"
Sep 17 15:59:03.979252 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.979235    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829\": container with ID starting with 45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829 not found: ID does not exist" containerID="45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"
Sep 17 15:59:03.979291 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979255    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"} err="failed to get container status \"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829\": rpc error: code = NotFound desc = could not find container \"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829\": container with ID starting with 45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829 not found: ID does not exist"
Sep 17 15:59:03.979291 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979267    2557 scope.go:117] "RemoveContainer" containerID="6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"
Sep 17 15:59:03.979484 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.979468    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2\": container with ID starting with 6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2 not found: ID does not exist" containerID="6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"
Sep 17 15:59:03.979521 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979491    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"} err="failed to get container status \"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2\": rpc error: code = NotFound desc = could not find container \"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2\": container with ID starting with 6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2 not found: ID does not exist"
Sep 17 15:59:03.979521 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979512    2557 scope.go:117] "RemoveContainer" containerID="24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"
Sep 17 15:59:03.979734 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.979716    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9\": container with ID starting with 24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9 not found: ID does not exist" containerID="24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"
Sep 17 15:59:03.979819 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979734    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"} err="failed to get container status \"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9\": rpc error: code = NotFound desc = could not find container \"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9\": container with ID starting with 24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9 not found: ID does not exist"
Sep 17 15:59:03.979819 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.979747    2557 scope.go:117] "RemoveContainer" containerID="2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"
Sep 17 15:59:03.980000 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.979983    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150\": container with ID starting with 2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150 not found: ID does not exist" containerID="2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"
Sep 17 15:59:03.980050 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980011    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"} err="failed to get container status \"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150\": rpc error: code = NotFound desc = could not find container \"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150\": container with ID starting with 2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150 not found: ID does not exist"
Sep 17 15:59:03.980050 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980025    2557 scope.go:117] "RemoveContainer" containerID="97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"
Sep 17 15:59:03.980271 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.980254    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541\": container with ID starting with 97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541 not found: ID does not exist" containerID="97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"
Sep 17 15:59:03.980319 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980279    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"} err="failed to get container status \"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541\": rpc error: code = NotFound desc = could not find container \"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541\": container with ID starting with 97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541 not found: ID does not exist"
Sep 17 15:59:03.980319 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980296    2557 scope.go:117] "RemoveContainer" containerID="0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"
Sep 17 15:59:03.980541 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:03.980527    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78\": container with ID starting with 0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78 not found: ID does not exist" containerID="0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"
Sep 17 15:59:03.980609 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980545    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"} err="failed to get container status \"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78\": rpc error: code = NotFound desc = could not find container \"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78\": container with ID starting with 0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78 not found: ID does not exist"
Sep 17 15:59:03.980609 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980559    2557 scope.go:117] "RemoveContainer" containerID="07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"
Sep 17 15:59:03.980792 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980777    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db"} err="failed to get container status \"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db\": rpc error: code = NotFound desc = could not find container \"07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db\": container with ID starting with 07be193c1366ee964bd701e39b74d09c44f99fe39210944439a022c1e5b0c8db not found: ID does not exist"
Sep 17 15:59:03.980834 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.980792    2557 scope.go:117] "RemoveContainer" containerID="45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"
Sep 17 15:59:03.981019 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981000    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829"} err="failed to get container status \"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829\": rpc error: code = NotFound desc = could not find container \"45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829\": container with ID starting with 45847cfc3ebad486d46b316fb509d286ec720d21cf5827f7eb27765f5e235829 not found: ID does not exist"
Sep 17 15:59:03.981068 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981019    2557 scope.go:117] "RemoveContainer" containerID="6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"
Sep 17 15:59:03.981235 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981220    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2"} err="failed to get container status \"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2\": rpc error: code = NotFound desc = could not find container \"6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2\": container with ID starting with 6cbac7c6f0fc9c856e6d0ca5c099f29870502d591bb72a85fff0a8e3d2c983a2 not found: ID does not exist"
Sep 17 15:59:03.981286 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981234    2557 scope.go:117] "RemoveContainer" containerID="24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"
Sep 17 15:59:03.981432 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981419    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9"} err="failed to get container status \"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9\": rpc error: code = NotFound desc = could not find container \"24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9\": container with ID starting with 24bc8ba991268f61105f1cabfba5a08cec4d25d9dbc9ef3c89397a9657249be9 not found: ID does not exist"
Sep 17 15:59:03.981481 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981431    2557 scope.go:117] "RemoveContainer" containerID="2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"
Sep 17 15:59:03.981661 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981642    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150"} err="failed to get container status \"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150\": rpc error: code = NotFound desc = could not find container \"2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150\": container with ID starting with 2bb3ec93ce67cbfe1dd8751bd4e0b4e038b8870d6e2d6329bc536f0ef1c21150 not found: ID does not exist"
Sep 17 15:59:03.981727 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981661    2557 scope.go:117] "RemoveContainer" containerID="97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"
Sep 17 15:59:03.981866 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981851    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541"} err="failed to get container status \"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541\": rpc error: code = NotFound desc = could not find container \"97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541\": container with ID starting with 97854b734c9ac1f135519a0056f532669861d669baf481f7f2c4ac20b7356541 not found: ID does not exist"
Sep 17 15:59:03.981906 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.981866    2557 scope.go:117] "RemoveContainer" containerID="0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"
Sep 17 15:59:03.982094 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:03.982072    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78"} err="failed to get container status \"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78\": rpc error: code = NotFound desc = could not find container \"0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78\": container with ID starting with 0be7d0c990140b4d1cbd5e32fd6b2c11c9752b63fcf69f0033970fdc4711ad78 not found: ID does not exist"
Sep 17 15:59:04.060799 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.060770    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:59:04.061089 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061076    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy"
Sep 17 15:59:04.061126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061091    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy"
Sep 17 15:59:04.061126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061105    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="prom-label-proxy"
Sep 17 15:59:04.061126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061112    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="prom-label-proxy"
Sep 17 15:59:04.061126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061119    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="alertmanager"
Sep 17 15:59:04.061126 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061125    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="alertmanager"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061132    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017d8d03-3f22-469b-bebe-58c6258404ac" containerName="registry"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061137    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="017d8d03-3f22-469b-bebe-58c6258404ac" containerName="registry"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061143    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-web"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061148    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-web"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061154    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="config-reloader"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061159    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="config-reloader"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061170    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-metric"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061175    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-metric"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061182    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="init-config-reloader"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061188    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="init-config-reloader"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061241    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-metric"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061251    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="prom-label-proxy"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061255    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="017d8d03-3f22-469b-bebe-58c6258404ac" containerName="registry"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061261    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="config-reloader"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061266    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="alertmanager"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061272    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy"
Sep 17 15:59:04.061279 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.061277    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" containerName="kube-rbac-proxy-web"
Sep 17 15:59:04.065092 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.065075    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.067905 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.067884    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:59:04.068036 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.068021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:59:04.068101 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.068085    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:59:04.069095 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.069074    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:59:04.069172 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.069156    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:59:04.069172 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.069168    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:59:04.072359 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.070482    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:59:04.078339 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.078322    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:59:04.078451 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.078324    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xxbbp\""
Sep 17 15:59:04.086640 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.086624    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:59:04.093229 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093343 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093343 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093531 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093505    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b6603c1-8c14-4e55-a970-83de991de4e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093655 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093671    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b6603c1-8c14-4e55-a970-83de991de4e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b6603c1-8c14-4e55-a970-83de991de4e7-config-out\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66vrw\" (UniqueName: \"kubernetes.io/projected/9b6603c1-8c14-4e55-a970-83de991de4e7-kube-api-access-66vrw\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093973 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093973 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-web-config\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093973 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b6603c1-8c14-4e55-a970-83de991de4e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.093973 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.093883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b6603c1-8c14-4e55-a970-83de991de4e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.094709 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.094692    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:59:04.194946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.194858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.194946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.194894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.194946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.194922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.194976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b6603c1-8c14-4e55-a970-83de991de4e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b6603c1-8c14-4e55-a970-83de991de4e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b6603c1-8c14-4e55-a970-83de991de4e7-config-out\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66vrw\" (UniqueName: \"kubernetes.io/projected/9b6603c1-8c14-4e55-a970-83de991de4e7-kube-api-access-66vrw\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-web-config\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195202    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b6603c1-8c14-4e55-a970-83de991de4e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195229    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b6603c1-8c14-4e55-a970-83de991de4e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.195476 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b6603c1-8c14-4e55-a970-83de991de4e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.196062 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.195862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b6603c1-8c14-4e55-a970-83de991de4e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.196221 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.196192    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b6603c1-8c14-4e55-a970-83de991de4e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198103 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198071    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198103 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198098    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198349 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b6603c1-8c14-4e55-a970-83de991de4e7-config-out\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198436 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198331    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198484 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198450    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198539 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198517    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.198921 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.198901    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.199185 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.199171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b6603c1-8c14-4e55-a970-83de991de4e7-web-config\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.199977 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.199962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b6603c1-8c14-4e55-a970-83de991de4e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.209161 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.209141    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66vrw\" (UniqueName: \"kubernetes.io/projected/9b6603c1-8c14-4e55-a970-83de991de4e7-kube-api-access-66vrw\") pod \"alertmanager-main-0\" (UID: \"9b6603c1-8c14-4e55-a970-83de991de4e7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.376116 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.376076    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:59:04.665497 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.665467    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:59:04.666509 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:59:04.666480    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6603c1_8c14_4e55_a970_83de991de4e7.slice/crio-5505c59097e437572f36e44842b479df9fa26a0dd2dceca5d5eadf09abcd062b WatchSource:0}: Error finding container 5505c59097e437572f36e44842b479df9fa26a0dd2dceca5d5eadf09abcd062b: Status 404 returned error can't find the container with id 5505c59097e437572f36e44842b479df9fa26a0dd2dceca5d5eadf09abcd062b
Sep 17 15:59:04.931665 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.931554    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b6603c1-8c14-4e55-a970-83de991de4e7" containerID="f71bc02648f8a6fae458de1784e652dd1e63412d23bc9efde660ff4596d54f6d" exitCode=0
Sep 17 15:59:04.931665 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.931643    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerDied","Data":"f71bc02648f8a6fae458de1784e652dd1e63412d23bc9efde660ff4596d54f6d"}
Sep 17 15:59:04.931846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:04.931732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"5505c59097e437572f36e44842b479df9fa26a0dd2dceca5d5eadf09abcd062b"}
Sep 17 15:59:05.028481 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.028453    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f02859-1691-4cd3-b67f-3b06f8e666cc" path="/var/lib/kubelet/pods/f9f02859-1691-4cd3-b67f-3b06f8e666cc/volumes"
Sep 17 15:59:05.939964 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.939930    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"8d14666d0713508b02c82b845a3ed84617461823fe9b0722986e2f64d7269846"}
Sep 17 15:59:05.940341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.939992    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"b6c25cfd407aee044d68bb7808a595f9e8b6c2f418861818a6aeb0d48c66b782"}
Sep 17 15:59:05.940341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.940022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"48b43587688f32e795a84a1ae7b459686026a4423dd84e9935e69f2159694ae3"}
Sep 17 15:59:05.940341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.940052    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"1dfe099874e8be64fb7a0b1abe4062c31ba4c9ad8f8eaa9921f3d718e2f22f35"}
Sep 17 15:59:05.940341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.940080    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"6eee6450802c753eaf451e7dd271e040a26ef455044dfb45e6f01012fe732656"}
Sep 17 15:59:05.940341 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.940106    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b6603c1-8c14-4e55-a970-83de991de4e7","Type":"ContainerStarted","Data":"d4455dabf9d59497c0f10fcfc552dd4b925050b977d285a29bf187b9897aeb70"}
Sep 17 15:59:05.982542 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:05.982480    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9655587730000001 podStartE2EDuration="1.982460728s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.901955ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:05.979234916 +0000 UTC m=+221.278379423" watchObservedRunningTime="2026-09-17 15:59:05.982460728 +0000 UTC m=+221.281605209"
Sep 17 15:59:06.238932 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.238829    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:59:06.239295 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.239262    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="prometheus" containerID="cri-o://dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b" gracePeriod=600
Sep 17 15:59:06.239413 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.239307    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e" gracePeriod=600
Sep 17 15:59:06.239413 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.239327    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-web" containerID="cri-o://0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e" gracePeriod=600
Sep 17 15:59:06.239413 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.239310    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="thanos-sidecar" containerID="cri-o://0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4" gracePeriod=600
Sep 17 15:59:06.239413 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.239269    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy" containerID="cri-o://1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f" gracePeriod=600
Sep 17 15:59:06.239643 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.239499    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="config-reloader" containerID="cri-o://b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8" gracePeriod=600
Sep 17 15:59:06.482021 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.481995    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:06.516211 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516122    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-web-config\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516211 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516174    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-kube-rbac-proxy\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516427 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516219    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516427 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516251    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-db\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516427 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516280    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z85c6\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-kube-api-access-z85c6\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516707 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516682    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516802 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516728    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-serving-certs-ca-bundle\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516802 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-2\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516802 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516794    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-tls\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516819    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-trusted-ca-bundle\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516845    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-thanos-prometheus-http-client-file\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516898    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-0\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516924    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-grpc-tls\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.516966 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516948    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-tls-assets\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.516977    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config-out\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517010    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517047    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-metrics-client-certs\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517076    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-metrics-client-ca\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517113    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-1\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517214 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517152    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-kubelet-serving-ca-bundle\") pod \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\" (UID: \"79c73b93-2548-4d1d-a158-1d9a7a0fefae\") "
Sep 17 15:59:06.517527 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517495    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.517769    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.518438    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.518824    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.519029    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.519093    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-kube-api-access-z85c6" (OuterVolumeSpecName: "kube-api-access-z85c6") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "kube-api-access-z85c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.519169    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.519412    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.519448    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:06.519510 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.519476    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.521485 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.521389    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:06.521485 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.521435    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.521485 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.521460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config" (OuterVolumeSpecName: "config") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.521730 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.521659    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.522043 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.522000    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.522043 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.522031    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.522176 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.522044    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config-out" (OuterVolumeSpecName: "config-out") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:06.522372 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.522357    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.523120 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.523087    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.532707 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.532679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-web-config" (OuterVolumeSpecName: "web-config") pod "79c73b93-2548-4d1d-a158-1d9a7a0fefae" (UID: "79c73b93-2548-4d1d-a158-1d9a7a0fefae"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:06.617785 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617748    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-metrics-client-ca\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.617785 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617777    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.617785 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617792    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617805    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-web-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617817    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-kube-rbac-proxy\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617828    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617841    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-db\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617852    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85c6\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-kube-api-access-z85c6\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617866    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617877    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617889    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617901    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617915    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617927    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617938    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c73b93-2548-4d1d-a158-1d9a7a0fefae-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617950    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-grpc-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617962    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c73b93-2548-4d1d-a158-1d9a7a0fefae-tls-assets\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617973    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c73b93-2548-4d1d-a158-1d9a7a0fefae-config-out\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617984    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.618056 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.617999    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c73b93-2548-4d1d-a158-1d9a7a0fefae-secret-metrics-client-certs\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 15:59:06.947803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947768    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e" exitCode=0
Sep 17 15:59:06.947803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947797    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f" exitCode=0
Sep 17 15:59:06.947803 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947806    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e" exitCode=0
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947817    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4" exitCode=0
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947825    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8" exitCode=0
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947835    2557 generic.go:356] "Generic (PLEG): container finished" podID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b" exitCode=0
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"}
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947861    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947954    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.947938    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"}
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.948068    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"}
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.948106    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"}
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.948144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"}
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.948177    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"}
Sep 17 15:59:06.948312 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.948240    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c73b93-2548-4d1d-a158-1d9a7a0fefae","Type":"ContainerDied","Data":"4fa1101a3731d2299a8865350774b9e06a42fc055e0dab7c712bf81632322df7"}
Sep 17 15:59:06.956579 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.956443    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:06.963750 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.963729    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:06.970396 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.970379    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:06.977002 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.976986    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:06.980848 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.980823    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:59:06.984226 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.984210    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:06.991296 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.991273    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:06.994625 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:06.993635    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:59:07.003397 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.003378    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.003657 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.003634    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.003746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.003664    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"} err="failed to get container status \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist"
Sep 17 15:59:07.003746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.003688    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.003960 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.003937    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.004016 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.003969    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"} err="failed to get container status \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist"
Sep 17 15:59:07.004016 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.003987    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.004241 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.004223    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.004282 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.004246    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"} err="failed to get container status \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist"
Sep 17 15:59:07.004282 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.004265    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.004518 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.004501    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.004583 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.004524    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"} err="failed to get container status \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist"
Sep 17 15:59:07.004583 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.004541    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.004864 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.004840    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.004905 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.004871    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"} err="failed to get container status \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist"
Sep 17 15:59:07.004905 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.004891    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.005216 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.005177    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.005305 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.005222    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"} err="failed to get container status \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist"
Sep 17 15:59:07.005305 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.005238    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.005507 ip-10-0-140-92 kubenswrapper[2557]: E0917 15:59:07.005478    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.005583 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.005503    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"} err="failed to get container status \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist"
Sep 17 15:59:07.005583 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.005521    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.005776 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.005755    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"} err="failed to get container status \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist"
Sep 17 15:59:07.005820 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.005778    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.006062 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006038    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"} err="failed to get container status \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist"
Sep 17 15:59:07.006062 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006061    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.006310 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006289    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"} err="failed to get container status \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist"
Sep 17 15:59:07.006310 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006309    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.006568 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006551    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"} err="failed to get container status \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist"
Sep 17 15:59:07.006639 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006568    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.006799 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006778    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"} err="failed to get container status \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist"
Sep 17 15:59:07.006799 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006801    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.006988 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006969    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"} err="failed to get container status \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist"
Sep 17 15:59:07.007043 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.006990    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.007243 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007223    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"} err="failed to get container status \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist"
Sep 17 15:59:07.007309 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007242    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.007485 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007466    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"} err="failed to get container status \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist"
Sep 17 15:59:07.007555 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007487    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.007745 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007728    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"} err="failed to get container status \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist"
Sep 17 15:59:07.007799 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007746    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.007981 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007964    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"} err="failed to get container status \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist"
Sep 17 15:59:07.008035 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.007982    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.008210 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008195    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"} err="failed to get container status \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist"
Sep 17 15:59:07.008210 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008208    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.008407 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008388    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"} err="failed to get container status \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist"
Sep 17 15:59:07.008475 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008408    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.008690 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008662    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"} err="failed to get container status \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist"
Sep 17 15:59:07.008690 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008681    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.008922 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008902    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"} err="failed to get container status \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist"
Sep 17 15:59:07.008922 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.008922    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.009128 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009105    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"} err="failed to get container status \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist"
Sep 17 15:59:07.009128 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009127    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.009336 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009313    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"} err="failed to get container status \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist"
Sep 17 15:59:07.009336 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009337    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.009568 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009550    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"} err="failed to get container status \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist"
Sep 17 15:59:07.009658 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009569    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.009842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009821    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"} err="failed to get container status \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist"
Sep 17 15:59:07.009842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.009839    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.012417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.012391    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"} err="failed to get container status \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist"
Sep 17 15:59:07.012417 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.012416    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.012946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.012923    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"} err="failed to get container status \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist"
Sep 17 15:59:07.012946 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.012944    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.013271 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.013236    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"} err="failed to get container status \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist"
Sep 17 15:59:07.013271 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.013271    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.013561 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.013543    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"} err="failed to get container status \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist"
Sep 17 15:59:07.013561 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.013563    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.013898 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.013879    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"} err="failed to get container status \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist"
Sep 17 15:59:07.013988 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.013900    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.014193 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014175    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"} err="failed to get container status \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist"
Sep 17 15:59:07.014246 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014194    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.014452 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014426    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"} err="failed to get container status \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist"
Sep 17 15:59:07.014525 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014453    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.014719 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014700    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"} err="failed to get container status \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist"
Sep 17 15:59:07.014719 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014718    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.014995 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014956    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"} err="failed to get container status \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist"
Sep 17 15:59:07.014995 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.014982    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.015196 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015174    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"} err="failed to get container status \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist"
Sep 17 15:59:07.015265 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015199    2557 scope.go:117] "RemoveContainer" containerID="2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"
Sep 17 15:59:07.015424 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015406    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e"} err="failed to get container status \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": rpc error: code = NotFound desc = could not find container \"2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e\": container with ID starting with 2ee8d6d4025b6d481f87dcd51eb08f06780b410edb59b17423d26c0c41654d9e not found: ID does not exist"
Sep 17 15:59:07.015508 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015426    2557 scope.go:117] "RemoveContainer" containerID="1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"
Sep 17 15:59:07.015682 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015659    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f"} err="failed to get container status \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": rpc error: code = NotFound desc = could not find container \"1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f\": container with ID starting with 1924d96c5b547d7247f527c85709d72443345983b85a597c71a4a855d05ddb1f not found: ID does not exist"
Sep 17 15:59:07.015789 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015688    2557 scope.go:117] "RemoveContainer" containerID="0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"
Sep 17 15:59:07.015950 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015922    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e"} err="failed to get container status \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": rpc error: code = NotFound desc = could not find container \"0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e\": container with ID starting with 0377c253a6b4d382f503793c0bef24836a11661094623ceb176baaaa9036029e not found: ID does not exist"
Sep 17 15:59:07.015950 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.015948    2557 scope.go:117] "RemoveContainer" containerID="0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"
Sep 17 15:59:07.016176 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016155    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4"} err="failed to get container status \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": rpc error: code = NotFound desc = could not find container \"0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4\": container with ID starting with 0c7b66574e8990fb9b7072fb18ef0061291063b27776f9f7f6a2faf00f31f6c4 not found: ID does not exist"
Sep 17 15:59:07.016176 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016174    2557 scope.go:117] "RemoveContainer" containerID="b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"
Sep 17 15:59:07.016455 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016437    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8"} err="failed to get container status \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": rpc error: code = NotFound desc = could not find container \"b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8\": container with ID starting with b06feab8666d4605a1e3d29c1450743139ea37cebfbdd02236354850bfe6f9c8 not found: ID does not exist"
Sep 17 15:59:07.016529 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016456    2557 scope.go:117] "RemoveContainer" containerID="dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"
Sep 17 15:59:07.016732 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016713    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b"} err="failed to get container status \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": rpc error: code = NotFound desc = could not find container \"dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b\": container with ID starting with dcfad6ecfa137629518055aec41135ec973f15f33aa1215988b32d05655b683b not found: ID does not exist"
Sep 17 15:59:07.016794 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016734    2557 scope.go:117] "RemoveContainer" containerID="54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"
Sep 17 15:59:07.016985 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.016956    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33"} err="failed to get container status \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": rpc error: code = NotFound desc = could not find container \"54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33\": container with ID starting with 54ae110d0d90ee5d10bcb42c943b25eb7b69ec0daa93e26b49e4e56ba394ac33 not found: ID does not exist"
Sep 17 15:59:07.026549 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.026527    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" path="/var/lib/kubelet/pods/79c73b93-2548-4d1d-a158-1d9a7a0fefae/volumes"
Sep 17 15:59:07.045364 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045339    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:59:07.045701 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045680    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-web"
Sep 17 15:59:07.045701 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045698    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-web"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045712    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045720    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045745    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="prometheus"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045753    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="prometheus"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045770    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-thanos"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045778    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-thanos"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045792    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="init-config-reloader"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045798    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="init-config-reloader"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045804    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="config-reloader"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045809    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="config-reloader"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045817    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="thanos-sidecar"
Sep 17 15:59:07.045842 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045825    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="thanos-sidecar"
Sep 17 15:59:07.046179 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045875    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-web"
Sep 17 15:59:07.046179 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045882    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="prometheus"
Sep 17 15:59:07.046179 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045888    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="thanos-sidecar"
Sep 17 15:59:07.046179 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045893    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy-thanos"
Sep 17 15:59:07.046179 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045899    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="kube-rbac-proxy"
Sep 17 15:59:07.046179 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.045905    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c73b93-2548-4d1d-a158-1d9a7a0fefae" containerName="config-reloader"
Sep 17 15:59:07.049504 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.049489    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.075609 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.075074    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:59:07.075609 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.075386    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:59:07.075609 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.075574    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:59:07.075846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.075782    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.075958    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.076217    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.076377    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.076716    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6g7qb\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.076766    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9fi2oo75t6ujn\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.076918    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.076962    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.077093    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:59:07.077462 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.077120    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:59:07.077852 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.077664    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:59:07.077852 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.077797    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:59:07.079539 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.079521    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:59:07.081614 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.081579    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:59:07.087904 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.087885    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:59:07.123355 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d943d705-859a-4e5a-9d9d-302d3a318bac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123355 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123565 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d943d705-859a-4e5a-9d9d-302d3a318bac-config-out\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123565 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123390    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123565 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123425    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123565 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123499    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fqh\" (UniqueName: \"kubernetes.io/projected/d943d705-859a-4e5a-9d9d-302d3a318bac-kube-api-access-t7fqh\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123565 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-config\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-web-config\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123623    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123739 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123734    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123910 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123749    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123910 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123910 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123812    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123910 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.123910 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.124047 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.124047 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.123937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.224846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.224773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d943d705-859a-4e5a-9d9d-302d3a318bac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.224846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.224803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.224846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.224819    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d943d705-859a-4e5a-9d9d-302d3a318bac-config-out\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.224846 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.224836    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.224860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.224888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fqh\" (UniqueName: \"kubernetes.io/projected/d943d705-859a-4e5a-9d9d-302d3a318bac-kube-api-access-t7fqh\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-config\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225037    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-web-config\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225112 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225386 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225346    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225821 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225821 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.225979 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.225824    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.227746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.226202    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.227746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.226475    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.227746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.226521    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.227746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.227419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.227746 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.227714    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.228215 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.227788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d943d705-859a-4e5a-9d9d-302d3a318bac-config-out\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.228215 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.228102    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-config\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.228305 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.228243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.228434 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.228387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.228980 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.228827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d943d705-859a-4e5a-9d9d-302d3a318bac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.229136 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.229112    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.229976 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.229621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.230077 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.230030    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.230320 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.230285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-web-config\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.230414 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.230340    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.230553 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.230537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.230585 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.230575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d943d705-859a-4e5a-9d9d-302d3a318bac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.230986 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.230969    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d943d705-859a-4e5a-9d9d-302d3a318bac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.235755 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.235737    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fqh\" (UniqueName: \"kubernetes.io/projected/d943d705-859a-4e5a-9d9d-302d3a318bac-kube-api-access-t7fqh\") pod \"prometheus-k8s-0\" (UID: \"d943d705-859a-4e5a-9d9d-302d3a318bac\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.377247 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.377213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:59:07.664691 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.664668    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:59:07.666258 ip-10-0-140-92 kubenswrapper[2557]: W0917 15:59:07.666232    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd943d705_859a_4e5a_9d9d_302d3a318bac.slice/crio-8ff9f30d3d6b4d8e1602a089343541d1071b73c8ee4fbffe64cca08d8e662c78 WatchSource:0}: Error finding container 8ff9f30d3d6b4d8e1602a089343541d1071b73c8ee4fbffe64cca08d8e662c78: Status 404 returned error can't find the container with id 8ff9f30d3d6b4d8e1602a089343541d1071b73c8ee4fbffe64cca08d8e662c78
Sep 17 15:59:07.955537 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.955455    2557 generic.go:356] "Generic (PLEG): container finished" podID="d943d705-859a-4e5a-9d9d-302d3a318bac" containerID="dad7ad3ade5d69e5fc757525ebac01d58cede2fc778dcab6143c7ef57e9cb8ab" exitCode=0
Sep 17 15:59:07.955537 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.955516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerDied","Data":"dad7ad3ade5d69e5fc757525ebac01d58cede2fc778dcab6143c7ef57e9cb8ab"}
Sep 17 15:59:07.955927 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:07.955577    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"8ff9f30d3d6b4d8e1602a089343541d1071b73c8ee4fbffe64cca08d8e662c78"}
Sep 17 15:59:08.963889 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:08.963848    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"cf8a185e1d5d70884cf906d4a6c8822daf803585114f3cb203b4854185c513b2"}
Sep 17 15:59:08.964248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:08.963951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"cb723db394dfee1e471f38403bb3d441de95e57f9bef8477c49118fa2b3818f6"}
Sep 17 15:59:08.964248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:08.963978    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"1cd0f17b9358855d0d43c62d25ac889fe0e6014eb431fb02203aa29fa1bca4c2"}
Sep 17 15:59:08.964248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:08.964010    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"3f468a659714c20c2cbcac77829b392eef376d43c18293730ea20ac6d9d81576"}
Sep 17 15:59:08.964248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:08.964037    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"0765aa8e9a79d613c46ec2f4b72fe30ed17405bee692c156ea96a852ad12901f"}
Sep 17 15:59:08.964248 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:08.964058    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d943d705-859a-4e5a-9d9d-302d3a318bac","Type":"ContainerStarted","Data":"fb49c5aee11cc2fcbe6e5551c35ec0caa067f54fefd40ef467c92a9b937aa919"}
Sep 17 15:59:09.010871 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:09.010811    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.993417019 podStartE2EDuration="2.010790259s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.37324ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:09.009522738 +0000 UTC m=+224.308667251" watchObservedRunningTime="2026-09-17 15:59:09.010790259 +0000 UTC m=+224.309934738"
Sep 17 15:59:12.377543 ip-10-0-140-92 kubenswrapper[2557]: I0917 15:59:12.377507    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:00:07.378150 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:07.378111    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:00:07.393460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:07.393436    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:00:08.198122 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:08.198091    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:00:26.905121 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:26.905086    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:00:26.910957 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:26.910935    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:00:26.911365 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:26.911345    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:00:26.914064 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:26.914048    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:00:26.916310 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:00:26.916289    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:02:45.478241 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.478166    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-c9sr2"]
Sep 17 16:02:45.480373 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.480356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.485862 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.485837    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:02:45.486054 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.486031    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:02:45.486735 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.486715    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:02:45.487563 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.487545    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2xpwv\""
Sep 17 16:02:45.495059 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.495037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-c9sr2"]
Sep 17 16:02:45.565652 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.565627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c07d1017-1953-4760-aaa7-f65677043405-data\") pod \"seaweedfs-64568bcd49-c9sr2\" (UID: \"c07d1017-1953-4760-aaa7-f65677043405\") " pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.565793 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.565670    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtb7l\" (UniqueName: \"kubernetes.io/projected/c07d1017-1953-4760-aaa7-f65677043405-kube-api-access-wtb7l\") pod \"seaweedfs-64568bcd49-c9sr2\" (UID: \"c07d1017-1953-4760-aaa7-f65677043405\") " pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.665995 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.665953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtb7l\" (UniqueName: \"kubernetes.io/projected/c07d1017-1953-4760-aaa7-f65677043405-kube-api-access-wtb7l\") pod \"seaweedfs-64568bcd49-c9sr2\" (UID: \"c07d1017-1953-4760-aaa7-f65677043405\") " pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.666150 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.666022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c07d1017-1953-4760-aaa7-f65677043405-data\") pod \"seaweedfs-64568bcd49-c9sr2\" (UID: \"c07d1017-1953-4760-aaa7-f65677043405\") " pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.666318 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.666303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c07d1017-1953-4760-aaa7-f65677043405-data\") pod \"seaweedfs-64568bcd49-c9sr2\" (UID: \"c07d1017-1953-4760-aaa7-f65677043405\") " pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.676404 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.676381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtb7l\" (UniqueName: \"kubernetes.io/projected/c07d1017-1953-4760-aaa7-f65677043405-kube-api-access-wtb7l\") pod \"seaweedfs-64568bcd49-c9sr2\" (UID: \"c07d1017-1953-4760-aaa7-f65677043405\") " pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:45.789656 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:45.789561    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:46.066752 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:46.066561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-c9sr2"]
Sep 17 16:02:46.069323 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:02:46.069295    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07d1017_1953_4760_aaa7_f65677043405.slice/crio-bc4aa37535c2038b37059edfce44da545495956193feca55801192caeb07d4b8 WatchSource:0}: Error finding container bc4aa37535c2038b37059edfce44da545495956193feca55801192caeb07d4b8: Status 404 returned error can't find the container with id bc4aa37535c2038b37059edfce44da545495956193feca55801192caeb07d4b8
Sep 17 16:02:46.070633 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:46.070612    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:02:46.781828 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:46.781795    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-c9sr2" event={"ID":"c07d1017-1953-4760-aaa7-f65677043405","Type":"ContainerStarted","Data":"bc4aa37535c2038b37059edfce44da545495956193feca55801192caeb07d4b8"}
Sep 17 16:02:49.795657 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:49.795557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-c9sr2" event={"ID":"c07d1017-1953-4760-aaa7-f65677043405","Type":"ContainerStarted","Data":"cc978b901171b33332360af1b6f14db03beb2d8dc61311078cb95f5cea0a8244"}
Sep 17 16:02:49.796083 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:49.795668    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:02:49.817083 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:49.817036    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-c9sr2" podStartSLOduration=1.418243252 podStartE2EDuration="4.81702375s" totalImagesPullingTime="3.398780498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:49.815374929 +0000 UTC m=+445.114519408" watchObservedRunningTime="2026-09-17 16:02:49.81702375 +0000 UTC m=+445.116168228"
Sep 17 16:02:55.802772 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:02:55.802741    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-c9sr2"
Sep 17 16:04:25.550664 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.550565    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-tcjqv"]
Sep 17 16:04:25.554002 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.553982    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.556628 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.556608    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:04:25.575271 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.575244    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-tcjqv"]
Sep 17 16:04:25.647093 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.647069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a02fa4-5377-4983-b72b-b47858f70163-model-cache\") pod \"s3-init-tcjqv\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") " pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.647195 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.647106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9w6\" (UniqueName: \"kubernetes.io/projected/c3a02fa4-5377-4983-b72b-b47858f70163-kube-api-access-4s9w6\") pod \"s3-init-tcjqv\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") " pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.747832 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.747809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a02fa4-5377-4983-b72b-b47858f70163-model-cache\") pod \"s3-init-tcjqv\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") " pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.747926 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.747837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9w6\" (UniqueName: \"kubernetes.io/projected/c3a02fa4-5377-4983-b72b-b47858f70163-kube-api-access-4s9w6\") pod \"s3-init-tcjqv\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") " pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.748140 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.748123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a02fa4-5377-4983-b72b-b47858f70163-model-cache\") pod \"s3-init-tcjqv\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") " pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.769066 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.769049    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9w6\" (UniqueName: \"kubernetes.io/projected/c3a02fa4-5377-4983-b72b-b47858f70163-kube-api-access-4s9w6\") pod \"s3-init-tcjqv\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") " pod="kserve/s3-init-tcjqv"
Sep 17 16:04:25.870850 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:25.870803    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tcjqv"
Sep 17 16:04:26.153436 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:26.153410    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-tcjqv"]
Sep 17 16:04:26.156103 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:04:26.156076    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3a02fa4_5377_4983_b72b_b47858f70163.slice/crio-10dbb7f6ad0d5549271f4259f0b6f295f823a4da0ddd2bc0415a4ca150b999eb WatchSource:0}: Error finding container 10dbb7f6ad0d5549271f4259f0b6f295f823a4da0ddd2bc0415a4ca150b999eb: Status 404 returned error can't find the container with id 10dbb7f6ad0d5549271f4259f0b6f295f823a4da0ddd2bc0415a4ca150b999eb
Sep 17 16:04:27.166334 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:27.166274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tcjqv" event={"ID":"c3a02fa4-5377-4983-b72b-b47858f70163","Type":"ContainerStarted","Data":"10dbb7f6ad0d5549271f4259f0b6f295f823a4da0ddd2bc0415a4ca150b999eb"}
Sep 17 16:04:31.006246 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:31.006222    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:04:31.184887 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:31.184859    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tcjqv" event={"ID":"c3a02fa4-5377-4983-b72b-b47858f70163","Type":"ContainerStarted","Data":"227091f43f7e2e27fd1e90e108add2f45909c5080a86526b8789c16d4fe89041"}
Sep 17 16:04:33.194375 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:33.194345    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3a02fa4-5377-4983-b72b-b47858f70163" containerID="227091f43f7e2e27fd1e90e108add2f45909c5080a86526b8789c16d4fe89041" exitCode=0
Sep 17 16:04:33.194801 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:33.194412    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tcjqv" event={"ID":"c3a02fa4-5377-4983-b72b-b47858f70163","Type":"ContainerDied","Data":"227091f43f7e2e27fd1e90e108add2f45909c5080a86526b8789c16d4fe89041"}
Sep 17 16:04:38.218952 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:38.218918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tcjqv" event={"ID":"c3a02fa4-5377-4983-b72b-b47858f70163","Type":"ContainerStarted","Data":"1f99897021587949d1c5ddef610f0be1e5fcd54152071ce0f25996771da68a8d"}
Sep 17 16:04:38.247330 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:04:38.247274    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-tcjqv" podStartSLOduration=2.295012959 podStartE2EDuration="13.247258153s" totalImagesPullingTime="9.221602398s" totalInitContainerRuntime="1.730642796s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:38.245911679 +0000 UTC m=+553.545056158" watchObservedRunningTime="2026-09-17 16:04:38.247258153 +0000 UTC m=+553.546402633"
Sep 17 16:05:26.934568 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:26.934542    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:05:26.937832 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:26.937810    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:05:26.939224 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:26.939205    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:05:26.942421 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:26.942405    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:05:42.463281 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:42.463252    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3a02fa4-5377-4983-b72b-b47858f70163" containerID="1f99897021587949d1c5ddef610f0be1e5fcd54152071ce0f25996771da68a8d" exitCode=0
Sep 17 16:05:42.463609 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:42.463317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tcjqv" event={"ID":"c3a02fa4-5377-4983-b72b-b47858f70163","Type":"ContainerDied","Data":"1f99897021587949d1c5ddef610f0be1e5fcd54152071ce0f25996771da68a8d"}
Sep 17 16:05:43.582006 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.581984    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tcjqv"
Sep 17 16:05:43.627516 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.627488    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s9w6\" (UniqueName: \"kubernetes.io/projected/c3a02fa4-5377-4983-b72b-b47858f70163-kube-api-access-4s9w6\") pod \"c3a02fa4-5377-4983-b72b-b47858f70163\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") "
Sep 17 16:05:43.627658 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.627570    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a02fa4-5377-4983-b72b-b47858f70163-model-cache\") pod \"c3a02fa4-5377-4983-b72b-b47858f70163\" (UID: \"c3a02fa4-5377-4983-b72b-b47858f70163\") "
Sep 17 16:05:43.629534 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.629513    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a02fa4-5377-4983-b72b-b47858f70163-kube-api-access-4s9w6" (OuterVolumeSpecName: "kube-api-access-4s9w6") pod "c3a02fa4-5377-4983-b72b-b47858f70163" (UID: "c3a02fa4-5377-4983-b72b-b47858f70163"). InnerVolumeSpecName "kube-api-access-4s9w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:43.648602 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.648568    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a02fa4-5377-4983-b72b-b47858f70163-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3a02fa4-5377-4983-b72b-b47858f70163" (UID: "c3a02fa4-5377-4983-b72b-b47858f70163"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:05:43.728577 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.728530    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9w6\" (UniqueName: \"kubernetes.io/projected/c3a02fa4-5377-4983-b72b-b47858f70163-kube-api-access-4s9w6\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:05:43.728577 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:43.728549    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a02fa4-5377-4983-b72b-b47858f70163-model-cache\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:05:44.472444 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:44.472416    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tcjqv"
Sep 17 16:05:44.472444 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:44.472439    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tcjqv" event={"ID":"c3a02fa4-5377-4983-b72b-b47858f70163","Type":"ContainerDied","Data":"10dbb7f6ad0d5549271f4259f0b6f295f823a4da0ddd2bc0415a4ca150b999eb"}
Sep 17 16:05:44.472714 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:44.472468    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10dbb7f6ad0d5549271f4259f0b6f295f823a4da0ddd2bc0415a4ca150b999eb"
Sep 17 16:05:57.856278 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.856243    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"]
Sep 17 16:05:57.856686 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.856534    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a02fa4-5377-4983-b72b-b47858f70163" containerName="s3-init"
Sep 17 16:05:57.856686 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.856545    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a02fa4-5377-4983-b72b-b47858f70163" containerName="s3-init"
Sep 17 16:05:57.856686 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.856558    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a02fa4-5377-4983-b72b-b47858f70163" containerName="download-hf-model"
Sep 17 16:05:57.856686 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.856564    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a02fa4-5377-4983-b72b-b47858f70163" containerName="download-hf-model"
Sep 17 16:05:57.856686 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.856637    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a02fa4-5377-4983-b72b-b47858f70163" containerName="s3-init"
Sep 17 16:05:57.859656 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.859639    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:57.863658 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.863630    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:05:57.863779 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.863726    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 17 16:05:57.863779 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.863733    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wx97n\""
Sep 17 16:05:57.863890 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.863797    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:05:57.863890 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.863844    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:05:57.864026 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.864004    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 17 16:05:57.864691 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.864666    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:05:57.877744 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.877713    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"]
Sep 17 16:05:57.931894 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.931867    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:57.932039 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.931904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drpkp\" (UniqueName: \"kubernetes.io/projected/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-api-access-drpkp\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:57.932039 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.931928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:57.932039 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.932024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:57.932178 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:57.932065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.019351 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.019325    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"]
Sep 17 16:05:58.022048 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.022031    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.024801 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.024786    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-48b6f-predictor-serving-cert\""
Sep 17 16:05:58.024890 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.024808    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-48b6f-kube-rbac-proxy-sar-config\""
Sep 17 16:05:58.032357 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.032340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.032438 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.032368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpkp\" (UniqueName: \"kubernetes.io/projected/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-api-access-drpkp\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.032438 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.032393    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.032508 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.032454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.032508 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.032502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.032662 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:05:58.032647    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 17 16:05:58.032715 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:05:58.032706    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls podName:eb7c0de5-24a4-4e31-a1e1-a50300dae572 nodeName:}" failed. No retries permitted until 2026-09-17 16:05:58.532690819 +0000 UTC m=+633.831835280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls") pod "isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" (UID: "eb7c0de5-24a4-4e31-a1e1-a50300dae572") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 17 16:05:58.032818 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.032803    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.033083 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.033063    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.033117 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.033088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.044283 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.044262    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"]
Sep 17 16:05:58.067425 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.067400    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpkp\" (UniqueName: \"kubernetes.io/projected/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-api-access-drpkp\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.133545 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.133477    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.133545 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.133519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e23660-3718-47a9-bdde-bdf34dd8c4c0-proxy-tls\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.133713 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.133611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrm7\" (UniqueName: \"kubernetes.io/projected/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-api-access-sxrm7\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.234143 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.234117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrm7\" (UniqueName: \"kubernetes.io/projected/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-api-access-sxrm7\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.234271 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.234165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.234271 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.234196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e23660-3718-47a9-bdde-bdf34dd8c4c0-proxy-tls\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.234858 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.234839    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.236299 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.236280    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e23660-3718-47a9-bdde-bdf34dd8c4c0-proxy-tls\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.247963 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.247944    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrm7\" (UniqueName: \"kubernetes.io/projected/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-api-access-sxrm7\") pod \"success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") " pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.256485 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.256464    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"]
Sep 17 16:05:58.259577 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.259564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.262263 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.262240    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 17 16:05:58.262342 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.262281    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 17 16:05:58.280121 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.280101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"]
Sep 17 16:05:58.331550 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.331528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:05:58.334518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.334500    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7j7v\" (UniqueName: \"kubernetes.io/projected/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-api-access-q7j7v\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.334613 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.334537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.334613 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.334554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.334613 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.334573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.334766 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.334685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.435487 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.435458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7j7v\" (UniqueName: \"kubernetes.io/projected/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-api-access-q7j7v\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.435623 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.435505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.435623 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.435547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.435623 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.435581    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.435728 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.435637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.436033 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.436013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.436268 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.436247    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.436372 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.436323    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.437898 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.437881    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.451815 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.451789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7j7v\" (UniqueName: \"kubernetes.io/projected/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-api-access-q7j7v\") pod \"isvc-xgboost-graph-predictor-555fc69d56-hq8gt\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.535968 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.535948    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.538174 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.538147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.569221 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.569199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:05:58.612063 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.612031    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"]
Sep 17 16:05:58.616935 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:05:58.616897    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e23660_3718_47a9_bdde_bdf34dd8c4c0.slice/crio-1a12e2921265cf15881b52da8dc50e6c17cb5dee2487d1fe26057fe516efcbf0 WatchSource:0}: Error finding container 1a12e2921265cf15881b52da8dc50e6c17cb5dee2487d1fe26057fe516efcbf0: Status 404 returned error can't find the container with id 1a12e2921265cf15881b52da8dc50e6c17cb5dee2487d1fe26057fe516efcbf0
Sep 17 16:05:58.687859 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.687795    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"]
Sep 17 16:05:58.691245 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.691229    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.696145 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.696123    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 17 16:05:58.696145 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.696138    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 17 16:05:58.711269 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.711240    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"]
Sep 17 16:05:58.737965 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.737941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q492\" (UniqueName: \"kubernetes.io/projected/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-api-access-7q492\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.738068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.738028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28463381-bdb3-4adf-9f12-e78f3b66abb1-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.738068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.738065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.738211 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.738090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.738211 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.738117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.769907 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.769888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:05:58.838704 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.838673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.838829 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.838723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.838829 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.838757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.838949 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:05:58.838829    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 16:05:58.838949 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:05:58.838899    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls podName:28463381-bdb3-4adf-9f12-e78f3b66abb1 nodeName:}" failed. No retries permitted until 2026-09-17 16:05:59.33887762 +0000 UTC m=+634.638022093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls") pod "isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" (UID: "28463381-bdb3-4adf-9f12-e78f3b66abb1") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 16:05:58.838949 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.838921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q492\" (UniqueName: \"kubernetes.io/projected/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-api-access-7q492\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.839101 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.839001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28463381-bdb3-4adf-9f12-e78f3b66abb1-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.839354 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.839330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28463381-bdb3-4adf-9f12-e78f3b66abb1-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.839456 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.839441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.839525 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.839485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.851090 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.851066    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"]
Sep 17 16:05:58.851679 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:58.851660    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q492\" (UniqueName: \"kubernetes.io/projected/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-api-access-7q492\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:58.855295 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:05:58.855276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a58a8f_f509_4f53_a4b0_e07c086a2e3e.slice/crio-4f61b5aadef2283b5e6fd2a8d6f1ec3872824ef9004e31a0f672fc8a5229e3ee WatchSource:0}: Error finding container 4f61b5aadef2283b5e6fd2a8d6f1ec3872824ef9004e31a0f672fc8a5229e3ee: Status 404 returned error can't find the container with id 4f61b5aadef2283b5e6fd2a8d6f1ec3872824ef9004e31a0f672fc8a5229e3ee
Sep 17 16:05:59.049739 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.049714    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"]
Sep 17 16:05:59.052005 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:05:59.051976    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7c0de5_24a4_4e31_a1e1_a50300dae572.slice/crio-b91ac87b96a7e3cb10057444825abb60ba6d2652ac15c777e8f8746fe0cb4c4e WatchSource:0}: Error finding container b91ac87b96a7e3cb10057444825abb60ba6d2652ac15c777e8f8746fe0cb4c4e: Status 404 returned error can't find the container with id b91ac87b96a7e3cb10057444825abb60ba6d2652ac15c777e8f8746fe0cb4c4e
Sep 17 16:05:59.344953 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.344376    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:59.349176 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.349122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:59.546062 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.545299    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerStarted","Data":"e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878"}
Sep 17 16:05:59.546062 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.545383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerStarted","Data":"b91ac87b96a7e3cb10057444825abb60ba6d2652ac15c777e8f8746fe0cb4c4e"}
Sep 17 16:05:59.554525 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.554492    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" event={"ID":"17e23660-3718-47a9-bdde-bdf34dd8c4c0","Type":"ContainerStarted","Data":"1a12e2921265cf15881b52da8dc50e6c17cb5dee2487d1fe26057fe516efcbf0"}
Sep 17 16:05:59.559719 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.559691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerStarted","Data":"2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950"}
Sep 17 16:05:59.559831 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.559752    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerStarted","Data":"4f61b5aadef2283b5e6fd2a8d6f1ec3872824ef9004e31a0f672fc8a5229e3ee"}
Sep 17 16:05:59.601213 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.601139    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:05:59.945331 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:05:59.945294    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"]
Sep 17 16:05:59.949986 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:05:59.949957    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28463381_bdb3_4adf_9f12_e78f3b66abb1.slice/crio-59b6568eb402751eb1692708d1a8e3d0e84a5236ace029f761733f9731739c5c WatchSource:0}: Error finding container 59b6568eb402751eb1692708d1a8e3d0e84a5236ace029f761733f9731739c5c: Status 404 returned error can't find the container with id 59b6568eb402751eb1692708d1a8e3d0e84a5236ace029f761733f9731739c5c
Sep 17 16:06:00.568685 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:00.568648    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerStarted","Data":"b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582"}
Sep 17 16:06:00.569136 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:00.568723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerStarted","Data":"59b6568eb402751eb1692708d1a8e3d0e84a5236ace029f761733f9731739c5c"}
Sep 17 16:06:03.587837 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:03.587761    2557 generic.go:356] "Generic (PLEG): container finished" podID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerID="2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950" exitCode=0
Sep 17 16:06:03.588262 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:03.587835    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerDied","Data":"2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950"}
Sep 17 16:06:04.596894 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:04.596811    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerID="e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878" exitCode=0
Sep 17 16:06:04.597298 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:04.596902    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerDied","Data":"e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878"}
Sep 17 16:06:04.599973 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:04.599949    2557 generic.go:356] "Generic (PLEG): container finished" podID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerID="b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582" exitCode=0
Sep 17 16:06:04.600079 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:04.599994    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerDied","Data":"b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582"}
Sep 17 16:06:26.720733 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:26.720652    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerStarted","Data":"201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719"}
Sep 17 16:06:26.722119 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:26.722087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" event={"ID":"17e23660-3718-47a9-bdde-bdf34dd8c4c0","Type":"ContainerStarted","Data":"8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f"}
Sep 17 16:06:26.723848 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:26.723826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerStarted","Data":"8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233"}
Sep 17 16:06:27.733158 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:27.733123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerStarted","Data":"2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03"}
Sep 17 16:06:30.753957 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.753919    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerStarted","Data":"61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3"}
Sep 17 16:06:30.754364 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.754138    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:06:30.754364 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.754326    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:06:30.755687 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.755660    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:06:30.756056 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.756030    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" event={"ID":"17e23660-3718-47a9-bdde-bdf34dd8c4c0","Type":"ContainerStarted","Data":"0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e"}
Sep 17 16:06:30.756175 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.756091    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:06:30.758103 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.758082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerStarted","Data":"0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57"}
Sep 17 16:06:30.758219 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.758134    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:06:30.758219 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.758167    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:06:30.759318 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.759289    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:06:30.760319 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.760301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerStarted","Data":"c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978"}
Sep 17 16:06:30.760564 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.760549    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:06:30.760627 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.760619    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:06:30.761462 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.761440    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:06:30.792136 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.787135    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podStartSLOduration=4.758561849 podStartE2EDuration="33.787118101s" totalImagesPullingTime="24.866754082s" totalInitContainerRuntime="4.16180217s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:30.784312876 +0000 UTC m=+666.083457372" watchObservedRunningTime="2026-09-17 16:06:30.787118101 +0000 UTC m=+666.086262581"
Sep 17 16:06:30.868223 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.868173    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podStartSLOduration=3.6784140929999998 podStartE2EDuration="32.868161158s" totalImagesPullingTime="24.947981328s" totalInitContainerRuntime="4.241765737s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:30.868040932 +0000 UTC m=+666.167185412" watchObservedRunningTime="2026-09-17 16:06:30.868161158 +0000 UTC m=+666.167305637"
Sep 17 16:06:30.868505 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.868478    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podStartSLOduration=2.5297027610000002 podStartE2EDuration="32.868470196s" totalImagesPullingTime="26.059106442s" totalInitContainerRuntime="4.279660993s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:30.827053026 +0000 UTC m=+666.126197504" watchObservedRunningTime="2026-09-17 16:06:30.868470196 +0000 UTC m=+666.167614669"
Sep 17 16:06:30.900488 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:30.900454    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podStartSLOduration=2.924084714 podStartE2EDuration="33.900442853s" totalImagesPullingTime="30.976358139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:30.897660829 +0000 UTC m=+666.196805329" watchObservedRunningTime="2026-09-17 16:06:30.900442853 +0000 UTC m=+666.199587332"
Sep 17 16:06:31.765004 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:31.764961    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:06:31.765476 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:31.764961    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:06:31.765476 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:31.765019    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:06:31.765583 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:31.765476    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:06:31.766156 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:31.766137    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 16:06:32.768716 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:32.768678    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 16:06:36.770359 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:36.770319    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:06:36.770795 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:36.770738    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:06:36.770865 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:36.770816    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:06:36.770865 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:36.770814    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:06:36.771216 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:36.771193    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:06:36.771321 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:36.771230    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:06:37.773125 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:37.773095    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:06:37.773562 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:37.773535    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 16:06:46.771816 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:46.771769    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:06:46.771816 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:46.771783    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:06:46.772211 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:46.771783    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:06:47.774328 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:47.774285    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 16:06:56.771530 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:56.771495    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:06:56.771980 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:56.771495    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:06:56.771980 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:56.771497    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:06:57.774177 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:06:57.774140    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 16:07:06.771813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:06.771774    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:07:06.772228 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:06.771768    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:07:06.772228 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:06.771861    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:07:07.773738 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:07.773698    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 16:07:16.774445 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:16.774407    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:07:16.774445 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:16.774388    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:07:16.774891 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:16.774456    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:07:17.774455 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:17.774428    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:07:26.771014 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:26.770976    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:07:26.771385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:26.771108    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:07:26.771385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:26.771181    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:07:36.771368 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:36.771326    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 16:07:36.771864 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:36.771326    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:07:36.771864 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:36.771730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:07:41.022437 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:41.022386    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:07:46.022775 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:46.022737    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:07:47.957408 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:47.957365    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"]
Sep 17 16:07:47.957836 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:47.957695    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container" containerID="cri-o://8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f" gracePeriod=30
Sep 17 16:07:47.957836 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:47.957746    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kube-rbac-proxy" containerID="cri-o://0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e" gracePeriod=30
Sep 17 16:07:48.097246 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.097210    2557 generic.go:356] "Generic (PLEG): container finished" podID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerID="0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e" exitCode=2
Sep 17 16:07:48.097431 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.097264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" event={"ID":"17e23660-3718-47a9-bdde-bdf34dd8c4c0","Type":"ContainerDied","Data":"0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e"}
Sep 17 16:07:48.157051 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.157014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"]
Sep 17 16:07:48.159775 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.159750    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.161933 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.161901    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dabd-predictor-serving-cert\""
Sep 17 16:07:48.162083 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.161969    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dabd-kube-rbac-proxy-sar-config\""
Sep 17 16:07:48.171410 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.171323    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"]
Sep 17 16:07:48.250216 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.250126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslns\" (UniqueName: \"kubernetes.io/projected/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-api-access-vslns\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.250216 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.250183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-proxy-tls\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.250216 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.250215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.351440 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.351400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.351645 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.351497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslns\" (UniqueName: \"kubernetes.io/projected/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-api-access-vslns\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.351645 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.351526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-proxy-tls\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.352135 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.352103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.354088 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.354061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-proxy-tls\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.360522 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.360496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslns\" (UniqueName: \"kubernetes.io/projected/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-api-access-vslns\") pod \"success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") " pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.470435 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.470391    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:48.760047 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:48.760011    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"]
Sep 17 16:07:48.760361 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:07:48.760335    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bb7c72_2aa2_44bd_a8bd_72da01cf0a1d.slice/crio-2189a690b06584f78cd0e9b881d5790d4e06d8324c4ec7b847d1361c6fe982d3 WatchSource:0}: Error finding container 2189a690b06584f78cd0e9b881d5790d4e06d8324c4ec7b847d1361c6fe982d3: Status 404 returned error can't find the container with id 2189a690b06584f78cd0e9b881d5790d4e06d8324c4ec7b847d1361c6fe982d3
Sep 17 16:07:49.103239 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:49.103149    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" event={"ID":"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d","Type":"ContainerStarted","Data":"c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711"}
Sep 17 16:07:49.103239 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:49.103215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" event={"ID":"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d","Type":"ContainerStarted","Data":"14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014"}
Sep 17 16:07:49.103755 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:49.103245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" event={"ID":"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d","Type":"ContainerStarted","Data":"2189a690b06584f78cd0e9b881d5790d4e06d8324c4ec7b847d1361c6fe982d3"}
Sep 17 16:07:49.103755 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:49.103298    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:49.129622 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:49.129559    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podStartSLOduration=1.129545619 podStartE2EDuration="1.129545619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:49.126654919 +0000 UTC m=+744.425799419" watchObservedRunningTime="2026-09-17 16:07:49.129545619 +0000 UTC m=+744.428690098"
Sep 17 16:07:50.107933 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.107906    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:50.109064 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.109033    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:07:50.692935 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.692911    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:07:50.773068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.773036    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxrm7\" (UniqueName: \"kubernetes.io/projected/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-api-access-sxrm7\") pod \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") "
Sep 17 16:07:50.773068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.773074    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-rbac-proxy-sar-config\") pod \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") "
Sep 17 16:07:50.773293 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.773097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e23660-3718-47a9-bdde-bdf34dd8c4c0-proxy-tls\") pod \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\" (UID: \"17e23660-3718-47a9-bdde-bdf34dd8c4c0\") "
Sep 17 16:07:50.773450 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.773424    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17e23660-3718-47a9-bdde-bdf34dd8c4c0" (UID: "17e23660-3718-47a9-bdde-bdf34dd8c4c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:50.775224 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.775202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-api-access-sxrm7" (OuterVolumeSpecName: "kube-api-access-sxrm7") pod "17e23660-3718-47a9-bdde-bdf34dd8c4c0" (UID: "17e23660-3718-47a9-bdde-bdf34dd8c4c0"). InnerVolumeSpecName "kube-api-access-sxrm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:50.775284 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.775236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e23660-3718-47a9-bdde-bdf34dd8c4c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17e23660-3718-47a9-bdde-bdf34dd8c4c0" (UID: "17e23660-3718-47a9-bdde-bdf34dd8c4c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:07:50.874493 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.874425    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrm7\" (UniqueName: \"kubernetes.io/projected/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-api-access-sxrm7\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:07:50.874493 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.874452    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e23660-3718-47a9-bdde-bdf34dd8c4c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:07:50.874493 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:50.874462    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e23660-3718-47a9-bdde-bdf34dd8c4c0-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:07:51.027124 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.027097    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:07:51.114525 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.114494    2557 generic.go:356] "Generic (PLEG): container finished" podID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerID="8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f" exitCode=0
Sep 17 16:07:51.114918 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.114550    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"
Sep 17 16:07:51.114918 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.114565    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" event={"ID":"17e23660-3718-47a9-bdde-bdf34dd8c4c0","Type":"ContainerDied","Data":"8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f"}
Sep 17 16:07:51.114918 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.114659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4" event={"ID":"17e23660-3718-47a9-bdde-bdf34dd8c4c0","Type":"ContainerDied","Data":"1a12e2921265cf15881b52da8dc50e6c17cb5dee2487d1fe26057fe516efcbf0"}
Sep 17 16:07:51.114918 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.114699    2557 scope.go:117] "RemoveContainer" containerID="0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e"
Sep 17 16:07:51.115211 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.115187    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:07:51.123137 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.123119    2557 scope.go:117] "RemoveContainer" containerID="8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f"
Sep 17 16:07:51.130009 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.129992    2557 scope.go:117] "RemoveContainer" containerID="0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e"
Sep 17 16:07:51.130273 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:07:51.130254    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e\": container with ID starting with 0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e not found: ID does not exist" containerID="0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e"
Sep 17 16:07:51.130332 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.130281    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e"} err="failed to get container status \"0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e\": rpc error: code = NotFound desc = could not find container \"0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e\": container with ID starting with 0416dc9456e8a5622e21ece89e2e30754cf3e3189e0a9d6e382b928a094fee2e not found: ID does not exist"
Sep 17 16:07:51.130332 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.130298    2557 scope.go:117] "RemoveContainer" containerID="8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f"
Sep 17 16:07:51.130556 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:07:51.130541    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f\": container with ID starting with 8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f not found: ID does not exist" containerID="8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f"
Sep 17 16:07:51.130616 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.130568    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f"} err="failed to get container status \"8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f\": rpc error: code = NotFound desc = could not find container \"8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f\": container with ID starting with 8a29a62e02c3db5b6f447da0359d0fab1bfd056f06b6279366e38bb48436080f not found: ID does not exist"
Sep 17 16:07:51.136556 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.136536    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"]
Sep 17 16:07:51.141905 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:51.141885    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48b6f-predictor-7bcc757b48-6r5h4"]
Sep 17 16:07:53.027512 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:53.027467    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" path="/var/lib/kubelet/pods/17e23660-3718-47a9-bdde-bdf34dd8c4c0/volumes"
Sep 17 16:07:56.119899 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:56.119874    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:07:56.120343 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:07:56.120293    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:08:06.120819 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:06.120777    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:08:16.120493 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:16.120453    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:08:26.120994 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:26.120950    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:08:28.694153 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.694107    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"]
Sep 17 16:08:28.694641 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.694343    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container" containerID="cri-o://8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233" gracePeriod=30
Sep 17 16:08:28.694641 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.694415    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kube-rbac-proxy" containerID="cri-o://0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57" gracePeriod=30
Sep 17 16:08:28.762155 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.762126    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"]
Sep 17 16:08:28.762367 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.762339    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" containerID="cri-o://201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719" gracePeriod=30
Sep 17 16:08:28.762442 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.762401    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kube-rbac-proxy" containerID="cri-o://61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3" gracePeriod=30
Sep 17 16:08:28.843607 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.843569    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"]
Sep 17 16:08:28.843834 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.843809    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container" containerID="cri-o://2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03" gracePeriod=30
Sep 17 16:08:28.843917 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:28.843837    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kube-rbac-proxy" containerID="cri-o://c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978" gracePeriod=30
Sep 17 16:08:29.019177 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019148    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"]
Sep 17 16:08:29.019513 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019502    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kube-rbac-proxy"
Sep 17 16:08:29.019556 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019515    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kube-rbac-proxy"
Sep 17 16:08:29.019556 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019533    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container"
Sep 17 16:08:29.019556 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019538    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container"
Sep 17 16:08:29.019665 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019587    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kserve-container"
Sep 17 16:08:29.019665 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.019612    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e23660-3718-47a9-bdde-bdf34dd8c4c0" containerName="kube-rbac-proxy"
Sep 17 16:08:29.021535 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.021520    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.029095 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:29.029072    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" is forbidden: User \"system:node:ip-10-0-140-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-92.ec2.internal' and this object" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.029200 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:29.029093    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"success-200-isvc-4f67a-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-140-92.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f67a-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 17 16:08:29.029253 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:29.029226    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"success-200-isvc-4f67a-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f67a-predictor-serving-cert\"" type="*v1.Secret"
Sep 17 16:08:29.037358 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.037335    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"]
Sep 17 16:08:29.194114 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.194080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.194284 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.194130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447xt\" (UniqueName: \"kubernetes.io/projected/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-api-access-447xt\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.194284 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.194155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-proxy-tls\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.275585 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.275509    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerID="61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3" exitCode=2
Sep 17 16:08:29.275757 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.275583    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerDied","Data":"61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3"}
Sep 17 16:08:29.277644 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.277619    2557 generic.go:356] "Generic (PLEG): container finished" podID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerID="0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57" exitCode=2
Sep 17 16:08:29.277972 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.277690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerDied","Data":"0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57"}
Sep 17 16:08:29.279842 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.279822    2557 generic.go:356] "Generic (PLEG): container finished" podID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerID="c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978" exitCode=2
Sep 17 16:08:29.279953 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.279900    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerDied","Data":"c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978"}
Sep 17 16:08:29.294949 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.294924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447xt\" (UniqueName: \"kubernetes.io/projected/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-api-access-447xt\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.295047 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.294969    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-proxy-tls\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.295096 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.295083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:29.313186 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:29.313157    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-447xt\" (UniqueName: \"kubernetes.io/projected/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-api-access-447xt\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:30.132004 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:30.131976    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f67a-predictor-serving-cert\""
Sep 17 16:08:30.137531 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:30.137508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-proxy-tls\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:30.296248 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:30.296215    2557 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/success-200-isvc-4f67a-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:08:30.296414 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:30.296308    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config podName:4db8d94f-aacf-43d1-aeb3-4760ccb0ad90 nodeName:}" failed. No retries permitted until 2026-09-17 16:08:30.796286642 +0000 UTC m=+786.095431119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config") pod "success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" (UID: "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 16:08:30.520609 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:30.520574    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f67a-kube-rbac-proxy-sar-config\""
Sep 17 16:08:30.804841 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:30.804766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:30.805346 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:30.805318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") " pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:30.831756 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:30.831736    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:31.022752 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.022703    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 16:08:31.117884 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.117805    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"]
Sep 17 16:08:31.121084 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:08:31.121060    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db8d94f_aacf_43d1_aeb3_4760ccb0ad90.slice/crio-ad5b90ed95b74ab68d57b06074e8c8aaf0cc671c6aac7d6ce44f7b9a623301e8 WatchSource:0}: Error finding container ad5b90ed95b74ab68d57b06074e8c8aaf0cc671c6aac7d6ce44f7b9a623301e8: Status 404 returned error can't find the container with id ad5b90ed95b74ab68d57b06074e8c8aaf0cc671c6aac7d6ce44f7b9a623301e8
Sep 17 16:08:31.290073 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.290035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" event={"ID":"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90","Type":"ContainerStarted","Data":"5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017"}
Sep 17 16:08:31.290479 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.290091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" event={"ID":"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90","Type":"ContainerStarted","Data":"e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7"}
Sep 17 16:08:31.290479 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.290116    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:31.290479 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.290146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" event={"ID":"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90","Type":"ContainerStarted","Data":"ad5b90ed95b74ab68d57b06074e8c8aaf0cc671c6aac7d6ce44f7b9a623301e8"}
Sep 17 16:08:31.290479 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.290165    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:31.291557 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.291532    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:08:31.765883 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.765847    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 17 16:08:31.766057 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.765848    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 16:08:31.766057 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:31.765848    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 16:08:32.294875 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:32.294836    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:08:32.980197 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:32.980173    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:08:33.011461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.011421    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podStartSLOduration=5.011404457 podStartE2EDuration="5.011404457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:31.319624805 +0000 UTC m=+786.618769298" watchObservedRunningTime="2026-09-17 16:08:33.011404457 +0000 UTC m=+788.310548935"
Sep 17 16:08:33.020792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.020771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7j7v\" (UniqueName: \"kubernetes.io/projected/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-api-access-q7j7v\") pod \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") "
Sep 17 16:08:33.020894 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.020811    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kserve-provision-location\") pod \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") "
Sep 17 16:08:33.020894 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.020860    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-rbac-proxy-sar-config\") pod \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") "
Sep 17 16:08:33.021001 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.020897    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-cabundle-cert\") pod \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") "
Sep 17 16:08:33.021001 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.020977    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-proxy-tls\") pod \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\" (UID: \"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e\") "
Sep 17 16:08:33.021231 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.021195    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" (UID: "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:33.021348 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.021287    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" (UID: "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:33.021348 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.021298    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" (UID: "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:33.022850 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.022826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-api-access-q7j7v" (OuterVolumeSpecName: "kube-api-access-q7j7v") pod "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" (UID: "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e"). InnerVolumeSpecName "kube-api-access-q7j7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:33.023092 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.023060    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" (UID: "a7a58a8f-f509-4f53-a4b0-e07c086a2e3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:33.122681 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.122572    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.122681 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.122619    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7j7v\" (UniqueName: \"kubernetes.io/projected/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-api-access-q7j7v\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.122681 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.122633    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kserve-provision-location\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.122681 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.122648    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.122681 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.122663    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e-cabundle-cert\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.300402 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.300368    2557 generic.go:356] "Generic (PLEG): container finished" podID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerID="2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03" exitCode=0
Sep 17 16:08:33.300783 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.300430    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"
Sep 17 16:08:33.300783 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.300449    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerDied","Data":"2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03"}
Sep 17 16:08:33.300783 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.300506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt" event={"ID":"a7a58a8f-f509-4f53-a4b0-e07c086a2e3e","Type":"ContainerDied","Data":"4f61b5aadef2283b5e6fd2a8d6f1ec3872824ef9004e31a0f672fc8a5229e3ee"}
Sep 17 16:08:33.300783 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.300532    2557 scope.go:117] "RemoveContainer" containerID="c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978"
Sep 17 16:08:33.308420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.308405    2557 scope.go:117] "RemoveContainer" containerID="2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03"
Sep 17 16:08:33.315449 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.315433    2557 scope.go:117] "RemoveContainer" containerID="2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950"
Sep 17 16:08:33.322582 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.322567    2557 scope.go:117] "RemoveContainer" containerID="c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978"
Sep 17 16:08:33.322833 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:33.322816    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978\": container with ID starting with c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978 not found: ID does not exist" containerID="c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978"
Sep 17 16:08:33.322895 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.322841    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978"} err="failed to get container status \"c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978\": rpc error: code = NotFound desc = could not find container \"c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978\": container with ID starting with c7a6653106506fce79aa8bd826b6dcd3048f398b08233c48d39b84e383654978 not found: ID does not exist"
Sep 17 16:08:33.322895 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.322856    2557 scope.go:117] "RemoveContainer" containerID="2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03"
Sep 17 16:08:33.323112 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:33.323093    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03\": container with ID starting with 2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03 not found: ID does not exist" containerID="2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03"
Sep 17 16:08:33.323160 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.323118    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03"} err="failed to get container status \"2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03\": rpc error: code = NotFound desc = could not find container \"2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03\": container with ID starting with 2057a18fa997f63e4d5978424160977b39ac64476cffbf5bf76a3b9390eddc03 not found: ID does not exist"
Sep 17 16:08:33.323160 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.323133    2557 scope.go:117] "RemoveContainer" containerID="2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950"
Sep 17 16:08:33.323381 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:33.323364    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950\": container with ID starting with 2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950 not found: ID does not exist" containerID="2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950"
Sep 17 16:08:33.323420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.323385    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950"} err="failed to get container status \"2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950\": rpc error: code = NotFound desc = could not find container \"2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950\": container with ID starting with 2c226587e0d8ada2925a3efeb9d8a4a30452b8d40950b4f23934b5c3b6990950 not found: ID does not exist"
Sep 17 16:08:33.339977 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.339957    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"]
Sep 17 16:08:33.341439 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.341421    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-hq8gt"]
Sep 17 16:08:33.689046 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.689024    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:08:33.727491 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.727463    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kserve-provision-location\") pod \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") "
Sep 17 16:08:33.727647 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.727517    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls\") pod \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") "
Sep 17 16:08:33.727647 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.727561    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-cabundle-cert\") pod \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") "
Sep 17 16:08:33.727647 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.727610    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drpkp\" (UniqueName: \"kubernetes.io/projected/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-api-access-drpkp\") pod \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") "
Sep 17 16:08:33.727647 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.727643    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-rbac-proxy-sar-config\") pod \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\" (UID: \"eb7c0de5-24a4-4e31-a1e1-a50300dae572\") "
Sep 17 16:08:33.728127 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.727994    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb7c0de5-24a4-4e31-a1e1-a50300dae572" (UID: "eb7c0de5-24a4-4e31-a1e1-a50300dae572"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:33.728251 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.728214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb7c0de5-24a4-4e31-a1e1-a50300dae572" (UID: "eb7c0de5-24a4-4e31-a1e1-a50300dae572"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:33.728251 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.728240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb7c0de5-24a4-4e31-a1e1-a50300dae572" (UID: "eb7c0de5-24a4-4e31-a1e1-a50300dae572"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:33.729962 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.729932    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-api-access-drpkp" (OuterVolumeSpecName: "kube-api-access-drpkp") pod "eb7c0de5-24a4-4e31-a1e1-a50300dae572" (UID: "eb7c0de5-24a4-4e31-a1e1-a50300dae572"). InnerVolumeSpecName "kube-api-access-drpkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:33.730123 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.730101    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb7c0de5-24a4-4e31-a1e1-a50300dae572" (UID: "eb7c0de5-24a4-4e31-a1e1-a50300dae572"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:33.829250 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.829224    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kserve-provision-location\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.829250 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.829247    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb7c0de5-24a4-4e31-a1e1-a50300dae572-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.829401 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.829258    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-cabundle-cert\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.829401 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.829266    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drpkp\" (UniqueName: \"kubernetes.io/projected/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-api-access-drpkp\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.829401 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.829275    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb7c0de5-24a4-4e31-a1e1-a50300dae572-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:33.939849 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:33.939827    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:08:34.030975 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.030951    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-rbac-proxy-sar-config\") pod \"28463381-bdb3-4adf-9f12-e78f3b66abb1\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") "
Sep 17 16:08:34.031090 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.030990    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q492\" (UniqueName: \"kubernetes.io/projected/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-api-access-7q492\") pod \"28463381-bdb3-4adf-9f12-e78f3b66abb1\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") "
Sep 17 16:08:34.031090 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.031017    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28463381-bdb3-4adf-9f12-e78f3b66abb1-kserve-provision-location\") pod \"28463381-bdb3-4adf-9f12-e78f3b66abb1\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") "
Sep 17 16:08:34.031090 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.031059    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-cabundle-cert\") pod \"28463381-bdb3-4adf-9f12-e78f3b66abb1\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") "
Sep 17 16:08:34.031090 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.031079    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls\") pod \"28463381-bdb3-4adf-9f12-e78f3b66abb1\" (UID: \"28463381-bdb3-4adf-9f12-e78f3b66abb1\") "
Sep 17 16:08:34.031341 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.031317    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28463381-bdb3-4adf-9f12-e78f3b66abb1" (UID: "28463381-bdb3-4adf-9f12-e78f3b66abb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:34.031397 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.031333    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28463381-bdb3-4adf-9f12-e78f3b66abb1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28463381-bdb3-4adf-9f12-e78f3b66abb1" (UID: "28463381-bdb3-4adf-9f12-e78f3b66abb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:34.031397 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.031386    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28463381-bdb3-4adf-9f12-e78f3b66abb1" (UID: "28463381-bdb3-4adf-9f12-e78f3b66abb1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:34.033042 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.033021    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28463381-bdb3-4adf-9f12-e78f3b66abb1" (UID: "28463381-bdb3-4adf-9f12-e78f3b66abb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:34.033134 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.033053    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-api-access-7q492" (OuterVolumeSpecName: "kube-api-access-7q492") pod "28463381-bdb3-4adf-9f12-e78f3b66abb1" (UID: "28463381-bdb3-4adf-9f12-e78f3b66abb1"). InnerVolumeSpecName "kube-api-access-7q492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:34.132393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.132339    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:34.132393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.132359    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q492\" (UniqueName: \"kubernetes.io/projected/28463381-bdb3-4adf-9f12-e78f3b66abb1-kube-api-access-7q492\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:34.132393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.132368    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28463381-bdb3-4adf-9f12-e78f3b66abb1-kserve-provision-location\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:34.132393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.132377    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28463381-bdb3-4adf-9f12-e78f3b66abb1-cabundle-cert\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:34.132393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.132385    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28463381-bdb3-4adf-9f12-e78f3b66abb1-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:08:34.306079 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.306048    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerID="201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719" exitCode=0
Sep 17 16:08:34.306464 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.306101    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"
Sep 17 16:08:34.306464 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.306121    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerDied","Data":"201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719"}
Sep 17 16:08:34.306464 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.306178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf" event={"ID":"eb7c0de5-24a4-4e31-a1e1-a50300dae572","Type":"ContainerDied","Data":"b91ac87b96a7e3cb10057444825abb60ba6d2652ac15c777e8f8746fe0cb4c4e"}
Sep 17 16:08:34.306464 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.306209    2557 scope.go:117] "RemoveContainer" containerID="61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3"
Sep 17 16:08:34.308313 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.308294    2557 generic.go:356] "Generic (PLEG): container finished" podID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerID="8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233" exitCode=0
Sep 17 16:08:34.308419 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.308348    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"
Sep 17 16:08:34.308419 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.308376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerDied","Data":"8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233"}
Sep 17 16:08:34.308525 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.308434    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9" event={"ID":"28463381-bdb3-4adf-9f12-e78f3b66abb1","Type":"ContainerDied","Data":"59b6568eb402751eb1692708d1a8e3d0e84a5236ace029f761733f9731739c5c"}
Sep 17 16:08:34.315471 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.315412    2557 scope.go:117] "RemoveContainer" containerID="201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719"
Sep 17 16:08:34.323026 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.323012    2557 scope.go:117] "RemoveContainer" containerID="e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878"
Sep 17 16:08:34.329978 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.329963    2557 scope.go:117] "RemoveContainer" containerID="61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3"
Sep 17 16:08:34.330214 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:34.330197    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3\": container with ID starting with 61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3 not found: ID does not exist" containerID="61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3"
Sep 17 16:08:34.330254 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.330219    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3"} err="failed to get container status \"61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3\": rpc error: code = NotFound desc = could not find container \"61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3\": container with ID starting with 61fd7612bea577313c8405fe472b1b99cd3ae4dd94cd3056f06ba167894ae1e3 not found: ID does not exist"
Sep 17 16:08:34.330254 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.330234    2557 scope.go:117] "RemoveContainer" containerID="201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719"
Sep 17 16:08:34.330486 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:34.330466    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719\": container with ID starting with 201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719 not found: ID does not exist" containerID="201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719"
Sep 17 16:08:34.330529 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.330490    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719"} err="failed to get container status \"201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719\": rpc error: code = NotFound desc = could not find container \"201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719\": container with ID starting with 201005f355ae3499a95f2727d8fdcbde83253712747a8ba1c36e46524e893719 not found: ID does not exist"
Sep 17 16:08:34.330529 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.330505    2557 scope.go:117] "RemoveContainer" containerID="e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878"
Sep 17 16:08:34.330778 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:34.330764    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878\": container with ID starting with e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878 not found: ID does not exist" containerID="e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878"
Sep 17 16:08:34.330816 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.330781    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878"} err="failed to get container status \"e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878\": rpc error: code = NotFound desc = could not find container \"e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878\": container with ID starting with e63ef24fb68f7f222b904ea576ed73058da2e9d0a9b86f2f22f0c166110b4878 not found: ID does not exist"
Sep 17 16:08:34.330816 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.330793    2557 scope.go:117] "RemoveContainer" containerID="0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57"
Sep 17 16:08:34.337451 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.337436    2557 scope.go:117] "RemoveContainer" containerID="8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233"
Sep 17 16:08:34.343984 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.343968    2557 scope.go:117] "RemoveContainer" containerID="b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582"
Sep 17 16:08:34.350850 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.350818    2557 scope.go:117] "RemoveContainer" containerID="0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57"
Sep 17 16:08:34.351164 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:34.351146    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57\": container with ID starting with 0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57 not found: ID does not exist" containerID="0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57"
Sep 17 16:08:34.351231 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.351169    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57"} err="failed to get container status \"0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57\": rpc error: code = NotFound desc = could not find container \"0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57\": container with ID starting with 0b9f0f68c3ce6a2a32cbd952b176d5cdeb3095ae3c1ea149c34c31de4f139a57 not found: ID does not exist"
Sep 17 16:08:34.351231 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.351184    2557 scope.go:117] "RemoveContainer" containerID="8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233"
Sep 17 16:08:34.351453 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:34.351433    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233\": container with ID starting with 8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233 not found: ID does not exist" containerID="8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233"
Sep 17 16:08:34.351518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.351461    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233"} err="failed to get container status \"8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233\": rpc error: code = NotFound desc = could not find container \"8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233\": container with ID starting with 8fa2397796c80b7e6b749dbe8bf7791c556e6762b2013c9ca7774513eaffa233 not found: ID does not exist"
Sep 17 16:08:34.351518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.351480    2557 scope.go:117] "RemoveContainer" containerID="b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582"
Sep 17 16:08:34.351767 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:08:34.351750    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582\": container with ID starting with b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582 not found: ID does not exist" containerID="b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582"
Sep 17 16:08:34.351839 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.351772    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582"} err="failed to get container status \"b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582\": rpc error: code = NotFound desc = could not find container \"b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582\": container with ID starting with b06651a1b9d05896e160ed9372ed5eb7cef5893c93c1bbc7dfdd6cf662b38582 not found: ID does not exist"
Sep 17 16:08:34.354286 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.354265    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"]
Sep 17 16:08:34.374033 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.374012    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7c5995c664-t9nq9"]
Sep 17 16:08:34.437385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.437368    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"]
Sep 17 16:08:34.438614 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:34.438584    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5fdd7b49c5-tcqdf"]
Sep 17 16:08:35.026921 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:35.026891    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" path="/var/lib/kubelet/pods/28463381-bdb3-4adf-9f12-e78f3b66abb1/volumes"
Sep 17 16:08:35.027363 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:35.027349    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" path="/var/lib/kubelet/pods/a7a58a8f-f509-4f53-a4b0-e07c086a2e3e/volumes"
Sep 17 16:08:35.028108 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:35.028092    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" path="/var/lib/kubelet/pods/eb7c0de5-24a4-4e31-a1e1-a50300dae572/volumes"
Sep 17 16:08:36.121531 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:36.121506    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:08:37.299049 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:37.299025    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:08:37.299474 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:37.299450    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:08:47.300434 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:47.300352    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:08:57.300184 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:08:57.300144    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:09:07.299650 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:09:07.299614    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:09:17.300557 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:09:17.300528    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:10:26.963407 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:10:26.963332    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:10:26.964457 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:10:26.964433    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:10:26.968207 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:10:26.968188    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:10:26.968966 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:10:26.968952    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:15:26.990777 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:15:26.990745    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:15:26.992213 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:15:26.992196    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:15:26.995207 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:15:26.995187    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:15:26.996575 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:15:26.996559    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:17:02.930283 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:02.930245    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"]
Sep 17 16:17:02.930815 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:02.930538    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" containerID="cri-o://14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014" gracePeriod=30
Sep 17 16:17:02.930815 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:02.930573    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kube-rbac-proxy" containerID="cri-o://c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711" gracePeriod=30
Sep 17 16:17:03.054380 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054349    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"]
Sep 17 16:17:03.054741 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054720    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container"
Sep 17 16:17:03.054741 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054737    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054749    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="storage-initializer"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054758    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="storage-initializer"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054768    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kube-rbac-proxy"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054775    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kube-rbac-proxy"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054783    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kube-rbac-proxy"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054790    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kube-rbac-proxy"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054800    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054807    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054820    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054828    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054841    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="storage-initializer"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054850    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="storage-initializer"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054867    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kube-rbac-proxy"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054875    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kube-rbac-proxy"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054882    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="storage-initializer"
Sep 17 16:17:03.054936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054889    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="storage-initializer"
Sep 17 16:17:03.055792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.054989    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kube-rbac-proxy"
Sep 17 16:17:03.055792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.055000    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kube-rbac-proxy"
Sep 17 16:17:03.055792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.055006    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kube-rbac-proxy"
Sep 17 16:17:03.055792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.055013    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="28463381-bdb3-4adf-9f12-e78f3b66abb1" containerName="kserve-container"
Sep 17 16:17:03.055792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.055022    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7c0de5-24a4-4e31-a1e1-a50300dae572" containerName="kserve-container"
Sep 17 16:17:03.055792 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.055032    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a58a8f-f509-4f53-a4b0-e07c086a2e3e" containerName="kserve-container"
Sep 17 16:17:03.057051 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.057033    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.059795 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.059770    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c0d2-predictor-serving-cert\""
Sep 17 16:17:03.059905 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.059796    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c0d2-kube-rbac-proxy-sar-config\""
Sep 17 16:17:03.070115 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.070092    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"]
Sep 17 16:17:03.174394 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.174351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f272975-7e98-408c-b98a-7ec39f09044e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.174571 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.174428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.174571 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.174471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4dr\" (UniqueName: \"kubernetes.io/projected/1f272975-7e98-408c-b98a-7ec39f09044e-kube-api-access-rx4dr\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.275501 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.275395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f272975-7e98-408c-b98a-7ec39f09044e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.275501 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.275461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.275501 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.275499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4dr\" (UniqueName: \"kubernetes.io/projected/1f272975-7e98-408c-b98a-7ec39f09044e-kube-api-access-rx4dr\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.275798 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:03.275573    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-serving-cert: secret "success-200-isvc-7c0d2-predictor-serving-cert" not found
Sep 17 16:17:03.275798 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:03.275676    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls podName:1f272975-7e98-408c-b98a-7ec39f09044e nodeName:}" failed. No retries permitted until 2026-09-17 16:17:03.775654184 +0000 UTC m=+1299.074798644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls") pod "success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" (UID: "1f272975-7e98-408c-b98a-7ec39f09044e") : secret "success-200-isvc-7c0d2-predictor-serving-cert" not found
Sep 17 16:17:03.276108 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.276090    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f272975-7e98-408c-b98a-7ec39f09044e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.286329 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.286308    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4dr\" (UniqueName: \"kubernetes.io/projected/1f272975-7e98-408c-b98a-7ec39f09044e-kube-api-access-rx4dr\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.338624 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.338578    2557 generic.go:356] "Generic (PLEG): container finished" podID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerID="c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711" exitCode=2
Sep 17 16:17:03.338791 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.338658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" event={"ID":"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d","Type":"ContainerDied","Data":"c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711"}
Sep 17 16:17:03.779947 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.779913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.782333 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.782306    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls\") pod \"success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") " pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:03.968494 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:03.968455    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:04.246950 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:04.246927    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"]
Sep 17 16:17:04.249871 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:17:04.249845    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f272975_7e98_408c_b98a_7ec39f09044e.slice/crio-e5959514867060597444923e180d86ec646b377de90f418ecbe6788d0c8d36a2 WatchSource:0}: Error finding container e5959514867060597444923e180d86ec646b377de90f418ecbe6788d0c8d36a2: Status 404 returned error can't find the container with id e5959514867060597444923e180d86ec646b377de90f418ecbe6788d0c8d36a2
Sep 17 16:17:04.344659 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:04.344626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" event={"ID":"1f272975-7e98-408c-b98a-7ec39f09044e","Type":"ContainerStarted","Data":"7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad"}
Sep 17 16:17:04.344766 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:04.344703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" event={"ID":"1f272975-7e98-408c-b98a-7ec39f09044e","Type":"ContainerStarted","Data":"e5959514867060597444923e180d86ec646b377de90f418ecbe6788d0c8d36a2"}
Sep 17 16:17:05.351393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:05.351350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" event={"ID":"1f272975-7e98-408c-b98a-7ec39f09044e","Type":"ContainerStarted","Data":"031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d"}
Sep 17 16:17:05.351905 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:05.351433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:05.381856 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:05.381784    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podStartSLOduration=2.381757171 podStartE2EDuration="2.381757171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:05.37875253 +0000 UTC m=+1300.677897019" watchObservedRunningTime="2026-09-17 16:17:05.381757171 +0000 UTC m=+1300.680901650"
Sep 17 16:17:06.174391 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.174364    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:17:06.300384 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.300286    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-rbac-proxy-sar-config\") pod \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") "
Sep 17 16:17:06.300384 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.300386    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-proxy-tls\") pod \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") "
Sep 17 16:17:06.300650 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.300426    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vslns\" (UniqueName: \"kubernetes.io/projected/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-api-access-vslns\") pod \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\" (UID: \"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d\") "
Sep 17 16:17:06.300650 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.300626    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" (UID: "e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:06.302460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.302434    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" (UID: "e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:06.302460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.302436    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-api-access-vslns" (OuterVolumeSpecName: "kube-api-access-vslns") pod "e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" (UID: "e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d"). InnerVolumeSpecName "kube-api-access-vslns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:06.356922 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.356891    2557 generic.go:356] "Generic (PLEG): container finished" podID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerID="14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014" exitCode=0
Sep 17 16:17:06.357385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.356949    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"
Sep 17 16:17:06.357385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.356969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" event={"ID":"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d","Type":"ContainerDied","Data":"14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014"}
Sep 17 16:17:06.357385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.357024    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" event={"ID":"e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d","Type":"ContainerDied","Data":"2189a690b06584f78cd0e9b881d5790d4e06d8324c4ec7b847d1361c6fe982d3"}
Sep 17 16:17:06.357385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.357050    2557 scope.go:117] "RemoveContainer" containerID="c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711"
Sep 17 16:17:06.357650 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.357435    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:06.358912 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.358884    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:17:06.365307 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.365270    2557 scope.go:117] "RemoveContainer" containerID="14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014"
Sep 17 16:17:06.372948 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.372930    2557 scope.go:117] "RemoveContainer" containerID="c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711"
Sep 17 16:17:06.373249 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:06.373229    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711\": container with ID starting with c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711 not found: ID does not exist" containerID="c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711"
Sep 17 16:17:06.373303 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.373257    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711"} err="failed to get container status \"c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711\": rpc error: code = NotFound desc = could not find container \"c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711\": container with ID starting with c487678f87e93abac802d03d7660201a17f3e6cb6c565979b0ed47230029a711 not found: ID does not exist"
Sep 17 16:17:06.373303 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.373273    2557 scope.go:117] "RemoveContainer" containerID="14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014"
Sep 17 16:17:06.373535 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:06.373517    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014\": container with ID starting with 14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014 not found: ID does not exist" containerID="14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014"
Sep 17 16:17:06.373615 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.373539    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014"} err="failed to get container status \"14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014\": rpc error: code = NotFound desc = could not find container \"14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014\": container with ID starting with 14ee34b6f0366c7d3a35944e5b67791a207cc25cf9472039e20c95ae5f6e5014 not found: ID does not exist"
Sep 17 16:17:06.401460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.401420    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vslns\" (UniqueName: \"kubernetes.io/projected/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-api-access-vslns\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:17:06.401460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.401450    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:17:06.401460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.401461    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:17:06.405245 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.405224    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"]
Sep 17 16:17:06.406581 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:06.406564    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv"]
Sep 17 16:17:07.026996 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:07.026953    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" path="/var/lib/kubelet/pods/e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d/volumes"
Sep 17 16:17:07.115892 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:07.115845    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:17:07.121033 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:07.121000    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dabd-predictor-55d56cfcbb-j58wv" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: i/o timeout"
Sep 17 16:17:07.362706 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:07.362619    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:17:12.367739 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:12.367712    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:17:12.368181 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:12.368145    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:17:22.368442 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:22.368400    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:17:32.368654 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:32.368614    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:17:42.369164 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:42.369066    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:17:43.467078 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.467036    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"]
Sep 17 16:17:43.467561 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.467273    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container" containerID="cri-o://e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7" gracePeriod=30
Sep 17 16:17:43.467561 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.467314    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kube-rbac-proxy" containerID="cri-o://5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017" gracePeriod=30
Sep 17 16:17:43.624631 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.624578    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"]
Sep 17 16:17:43.624951 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.624937    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container"
Sep 17 16:17:43.624951 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.624951    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container"
Sep 17 16:17:43.625073 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.624979    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kube-rbac-proxy"
Sep 17 16:17:43.625073 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.624987    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kube-rbac-proxy"
Sep 17 16:17:43.625073 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.625040    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kube-rbac-proxy"
Sep 17 16:17:43.625073 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.625048    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bb7c72-2aa2-44bd-a8bd-72da01cf0a1d" containerName="kserve-container"
Sep 17 16:17:43.628125 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.628101    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.630814 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.630790    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-48574-predictor-serving-cert\""
Sep 17 16:17:43.630961 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.630824    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-48574-kube-rbac-proxy-sar-config\""
Sep 17 16:17:43.639891 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.639855    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"]
Sep 17 16:17:43.718002 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.717898    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56pfg\" (UniqueName: \"kubernetes.io/projected/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-api-access-56pfg\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.718002 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.717966    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.718212 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.718014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.818968 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.818926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.819157 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.818993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.819157 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.819028    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56pfg\" (UniqueName: \"kubernetes.io/projected/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-api-access-56pfg\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.819157 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:43.819077    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-48574-predictor-serving-cert: secret "success-200-isvc-48574-predictor-serving-cert" not found
Sep 17 16:17:43.819290 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:43.819190    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls podName:5f0ccf71-53ce-4105-a9bc-67974be20b8d nodeName:}" failed. No retries permitted until 2026-09-17 16:17:44.319169632 +0000 UTC m=+1339.618314089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls") pod "success-200-isvc-48574-predictor-69dd69f964-fv7vm" (UID: "5f0ccf71-53ce-4105-a9bc-67974be20b8d") : secret "success-200-isvc-48574-predictor-serving-cert" not found
Sep 17 16:17:43.819756 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.819738    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:43.831661 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:43.831615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56pfg\" (UniqueName: \"kubernetes.io/projected/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-api-access-56pfg\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:44.322216 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:44.322174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:44.324840 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:44.324809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls\") pod \"success-200-isvc-48574-predictor-69dd69f964-fv7vm\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") " pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:44.519545 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:44.519508    2557 generic.go:356] "Generic (PLEG): container finished" podID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerID="5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017" exitCode=2
Sep 17 16:17:44.519934 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:44.519581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" event={"ID":"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90","Type":"ContainerDied","Data":"5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017"}
Sep 17 16:17:44.538887 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:44.538850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:44.816407 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:44.816369    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"]
Sep 17 16:17:44.818873 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:17:44.818844    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0ccf71_53ce_4105_a9bc_67974be20b8d.slice/crio-5cd575b62d72b837f2658fb1e71b933a31baafe0beaa0e90d6a9daccdbd40712 WatchSource:0}: Error finding container 5cd575b62d72b837f2658fb1e71b933a31baafe0beaa0e90d6a9daccdbd40712: Status 404 returned error can't find the container with id 5cd575b62d72b837f2658fb1e71b933a31baafe0beaa0e90d6a9daccdbd40712
Sep 17 16:17:45.525886 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.525852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" event={"ID":"5f0ccf71-53ce-4105-a9bc-67974be20b8d","Type":"ContainerStarted","Data":"146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88"}
Sep 17 16:17:45.526253 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.525915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" event={"ID":"5f0ccf71-53ce-4105-a9bc-67974be20b8d","Type":"ContainerStarted","Data":"175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46"}
Sep 17 16:17:45.526253 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.525937    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" event={"ID":"5f0ccf71-53ce-4105-a9bc-67974be20b8d","Type":"ContainerStarted","Data":"5cd575b62d72b837f2658fb1e71b933a31baafe0beaa0e90d6a9daccdbd40712"}
Sep 17 16:17:45.526253 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.525958    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:45.526253 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.525985    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:45.527257 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.527226    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:17:45.549677 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:45.549622    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podStartSLOduration=2.549580391 podStartE2EDuration="2.549580391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:45.548703392 +0000 UTC m=+1340.847847897" watchObservedRunningTime="2026-09-17 16:17:45.549580391 +0000 UTC m=+1340.848724873"
Sep 17 16:17:46.529887 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.529852    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:17:46.819870 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.819848    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:17:46.842934 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.842912    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config\") pod \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") "
Sep 17 16:17:46.843070 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.842942    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-proxy-tls\") pod \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") "
Sep 17 16:17:46.843070 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.843014    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-447xt\" (UniqueName: \"kubernetes.io/projected/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-api-access-447xt\") pod \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\" (UID: \"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90\") "
Sep 17 16:17:46.843328 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.843289    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" (UID: "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:46.845133 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.845101    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" (UID: "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:46.845496 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.845468    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-api-access-447xt" (OuterVolumeSpecName: "kube-api-access-447xt") pod "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" (UID: "4db8d94f-aacf-43d1-aeb3-4760ccb0ad90"). InnerVolumeSpecName "kube-api-access-447xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:46.944421 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.944374    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447xt\" (UniqueName: \"kubernetes.io/projected/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-api-access-447xt\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:17:46.944421 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.944412    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:17:46.944421 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:46.944426    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:17:47.536054 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.536018    2557 generic.go:356] "Generic (PLEG): container finished" podID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerID="e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7" exitCode=0
Sep 17 16:17:47.536518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.536092    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"
Sep 17 16:17:47.536518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.536105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" event={"ID":"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90","Type":"ContainerDied","Data":"e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7"}
Sep 17 16:17:47.536518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.536171    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v" event={"ID":"4db8d94f-aacf-43d1-aeb3-4760ccb0ad90","Type":"ContainerDied","Data":"ad5b90ed95b74ab68d57b06074e8c8aaf0cc671c6aac7d6ce44f7b9a623301e8"}
Sep 17 16:17:47.536518 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.536197    2557 scope.go:117] "RemoveContainer" containerID="5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017"
Sep 17 16:17:47.545785 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.545758    2557 scope.go:117] "RemoveContainer" containerID="e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7"
Sep 17 16:17:47.555359 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.555335    2557 scope.go:117] "RemoveContainer" containerID="5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017"
Sep 17 16:17:47.555745 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:47.555725    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017\": container with ID starting with 5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017 not found: ID does not exist" containerID="5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017"
Sep 17 16:17:47.555812 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.555756    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017"} err="failed to get container status \"5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017\": rpc error: code = NotFound desc = could not find container \"5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017\": container with ID starting with 5145643e2a366868e2cafa6e819036dc57ea7ddff6c126d5febf971e60639017 not found: ID does not exist"
Sep 17 16:17:47.555812 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.555777    2557 scope.go:117] "RemoveContainer" containerID="e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7"
Sep 17 16:17:47.556059 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:17:47.556041    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7\": container with ID starting with e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7 not found: ID does not exist" containerID="e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7"
Sep 17 16:17:47.556101 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.556066    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7"} err="failed to get container status \"e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7\": rpc error: code = NotFound desc = could not find container \"e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7\": container with ID starting with e513924a4db7ad26163088f7dc1a53606651c98d1cf5e4ff7da56343b60bcfa7 not found: ID does not exist"
Sep 17 16:17:47.558727 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.558691    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"]
Sep 17 16:17:47.563580 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:47.563550    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f67a-predictor-86bc4cfbd8-kp27v"]
Sep 17 16:17:49.027811 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:49.027775    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" path="/var/lib/kubelet/pods/4db8d94f-aacf-43d1-aeb3-4760ccb0ad90/volumes"
Sep 17 16:17:51.534919 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:51.534891    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:17:51.535367 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:51.535339    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:17:52.368743 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:17:52.368712    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:18:01.535649 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:01.535585    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:18:11.535578 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:11.535539    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:18:13.341036 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.341001    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"]
Sep 17 16:18:13.341540 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.341206    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container" containerID="cri-o://7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad" gracePeriod=30
Sep 17 16:18:13.341540 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.341253    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kube-rbac-proxy" containerID="cri-o://031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d" gracePeriod=30
Sep 17 16:18:13.576898 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.576861    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"]
Sep 17 16:18:13.577189 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.577178    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kube-rbac-proxy"
Sep 17 16:18:13.577235 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.577191    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kube-rbac-proxy"
Sep 17 16:18:13.577235 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.577201    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container"
Sep 17 16:18:13.577235 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.577207    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container"
Sep 17 16:18:13.577337 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.577279    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kserve-container"
Sep 17 16:18:13.577337 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.577289    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db8d94f-aacf-43d1-aeb3-4760ccb0ad90" containerName="kube-rbac-proxy"
Sep 17 16:18:13.580225 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.580209    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.583232 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.583208    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1e5cb-kube-rbac-proxy-sar-config\""
Sep 17 16:18:13.583823 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.583809    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1e5cb-predictor-serving-cert\""
Sep 17 16:18:13.595295 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.595245    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"]
Sep 17 16:18:13.650687 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.650647    2557 generic.go:356] "Generic (PLEG): container finished" podID="1f272975-7e98-408c-b98a-7ec39f09044e" containerID="031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d" exitCode=2
Sep 17 16:18:13.650828 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.650700    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" event={"ID":"1f272975-7e98-408c-b98a-7ec39f09044e","Type":"ContainerDied","Data":"031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d"}
Sep 17 16:18:13.654953 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.654929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8f143e-2616-4647-8996-2b5e7b4f03c1-proxy-tls\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.655072 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.655047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzg7h\" (UniqueName: \"kubernetes.io/projected/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-api-access-jzg7h\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.655145 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.655118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.755714 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.755680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.755841 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.755723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8f143e-2616-4647-8996-2b5e7b4f03c1-proxy-tls\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.755841 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.755800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzg7h\" (UniqueName: \"kubernetes.io/projected/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-api-access-jzg7h\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.756383 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.756363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.758087 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.758066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8f143e-2616-4647-8996-2b5e7b4f03c1-proxy-tls\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.775637 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.775613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzg7h\" (UniqueName: \"kubernetes.io/projected/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-api-access-jzg7h\") pod \"success-200-isvc-1e5cb-predictor-5d6b764d8-286f9\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") " pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:13.890928 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:13.890857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:14.174214 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:14.174183    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"]
Sep 17 16:18:14.175887 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:18:14.175859    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8f143e_2616_4647_8996_2b5e7b4f03c1.slice/crio-ff0a5a1c9537c8846c3f158b767591373bbc66df5b304a6a62413adad04d8a68 WatchSource:0}: Error finding container ff0a5a1c9537c8846c3f158b767591373bbc66df5b304a6a62413adad04d8a68: Status 404 returned error can't find the container with id ff0a5a1c9537c8846c3f158b767591373bbc66df5b304a6a62413adad04d8a68
Sep 17 16:18:14.656912 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:14.656876    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" event={"ID":"9b8f143e-2616-4647-8996-2b5e7b4f03c1","Type":"ContainerStarted","Data":"06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765"}
Sep 17 16:18:14.657276 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:14.656944    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" event={"ID":"9b8f143e-2616-4647-8996-2b5e7b4f03c1","Type":"ContainerStarted","Data":"900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30"}
Sep 17 16:18:14.657276 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:14.656972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" event={"ID":"9b8f143e-2616-4647-8996-2b5e7b4f03c1","Type":"ContainerStarted","Data":"ff0a5a1c9537c8846c3f158b767591373bbc66df5b304a6a62413adad04d8a68"}
Sep 17 16:18:14.657276 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:14.657003    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:14.682971 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:14.682924    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podStartSLOduration=1.682909319 podStartE2EDuration="1.682909319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:14.680796526 +0000 UTC m=+1369.979941039" watchObservedRunningTime="2026-09-17 16:18:14.682909319 +0000 UTC m=+1369.982053798"
Sep 17 16:18:15.663947 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:15.663910    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:15.666003 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:15.665834    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:18:16.667021 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:16.666974    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:18:16.979122 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:16.979101    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:18:17.085147 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.085118    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f272975-7e98-408c-b98a-7ec39f09044e-kube-rbac-proxy-sar-config\") pod \"1f272975-7e98-408c-b98a-7ec39f09044e\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") "
Sep 17 16:18:17.085147 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.085150    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls\") pod \"1f272975-7e98-408c-b98a-7ec39f09044e\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") "
Sep 17 16:18:17.085328 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.085168    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx4dr\" (UniqueName: \"kubernetes.io/projected/1f272975-7e98-408c-b98a-7ec39f09044e-kube-api-access-rx4dr\") pod \"1f272975-7e98-408c-b98a-7ec39f09044e\" (UID: \"1f272975-7e98-408c-b98a-7ec39f09044e\") "
Sep 17 16:18:17.085461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.085441    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f272975-7e98-408c-b98a-7ec39f09044e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f272975-7e98-408c-b98a-7ec39f09044e" (UID: "1f272975-7e98-408c-b98a-7ec39f09044e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:17.087189 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.087154    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f272975-7e98-408c-b98a-7ec39f09044e" (UID: "1f272975-7e98-408c-b98a-7ec39f09044e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:17.087308 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.087249    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f272975-7e98-408c-b98a-7ec39f09044e-kube-api-access-rx4dr" (OuterVolumeSpecName: "kube-api-access-rx4dr") pod "1f272975-7e98-408c-b98a-7ec39f09044e" (UID: "1f272975-7e98-408c-b98a-7ec39f09044e"). InnerVolumeSpecName "kube-api-access-rx4dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:17.186768 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.186738    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f272975-7e98-408c-b98a-7ec39f09044e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:18:17.186768 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.186763    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f272975-7e98-408c-b98a-7ec39f09044e-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:18:17.186931 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.186780    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4dr\" (UniqueName: \"kubernetes.io/projected/1f272975-7e98-408c-b98a-7ec39f09044e-kube-api-access-rx4dr\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:18:17.672609 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.672555    2557 generic.go:356] "Generic (PLEG): container finished" podID="1f272975-7e98-408c-b98a-7ec39f09044e" containerID="7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad" exitCode=0
Sep 17 16:18:17.672986 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.672625    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"
Sep 17 16:18:17.672986 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.672629    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" event={"ID":"1f272975-7e98-408c-b98a-7ec39f09044e","Type":"ContainerDied","Data":"7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad"}
Sep 17 16:18:17.672986 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.672689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl" event={"ID":"1f272975-7e98-408c-b98a-7ec39f09044e","Type":"ContainerDied","Data":"e5959514867060597444923e180d86ec646b377de90f418ecbe6788d0c8d36a2"}
Sep 17 16:18:17.672986 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.672716    2557 scope.go:117] "RemoveContainer" containerID="031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d"
Sep 17 16:18:17.681291 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.681274    2557 scope.go:117] "RemoveContainer" containerID="7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad"
Sep 17 16:18:17.688419 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.688404    2557 scope.go:117] "RemoveContainer" containerID="031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d"
Sep 17 16:18:17.688738 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:18:17.688719    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d\": container with ID starting with 031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d not found: ID does not exist" containerID="031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d"
Sep 17 16:18:17.688795 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.688747    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d"} err="failed to get container status \"031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d\": rpc error: code = NotFound desc = could not find container \"031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d\": container with ID starting with 031f3e355aee8c8e4b20d0bb42c1dc35a2b2e8cfc910a092a1cd55d4a6ad662d not found: ID does not exist"
Sep 17 16:18:17.688795 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.688764    2557 scope.go:117] "RemoveContainer" containerID="7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad"
Sep 17 16:18:17.688991 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:18:17.688973    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad\": container with ID starting with 7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad not found: ID does not exist" containerID="7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad"
Sep 17 16:18:17.689035 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.688996    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad"} err="failed to get container status \"7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad\": rpc error: code = NotFound desc = could not find container \"7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad\": container with ID starting with 7526a05a9573fd5dc708d1c83c2b76d867c467d1c328944ef4c101bb1a42dbad not found: ID does not exist"
Sep 17 16:18:17.700378 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.700354    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"]
Sep 17 16:18:17.702316 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:17.702297    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c0d2-predictor-bbb4bd85b-kv5kl"]
Sep 17 16:18:19.027328 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:19.027298    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" path="/var/lib/kubelet/pods/1f272975-7e98-408c-b98a-7ec39f09044e/volumes"
Sep 17 16:18:21.536260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:21.536221    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:18:21.671406 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:21.671376    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:18:21.671783 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:21.671756    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:18:31.536471 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:31.536441    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:18:31.671872 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:31.671829    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:18:41.672722 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:41.672682    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:18:51.671956 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:51.671915    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:18:53.825448 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.825416    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"]
Sep 17 16:18:53.825877 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.825640    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container" containerID="cri-o://175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46" gracePeriod=30
Sep 17 16:18:53.825877 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.825691    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kube-rbac-proxy" containerID="cri-o://146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88" gracePeriod=30
Sep 17 16:18:53.975447 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975420    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"]
Sep 17 16:18:53.975830 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975815    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kube-rbac-proxy"
Sep 17 16:18:53.975921 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975833    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kube-rbac-proxy"
Sep 17 16:18:53.975921 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975845    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container"
Sep 17 16:18:53.975921 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975853    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container"
Sep 17 16:18:53.976076 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975934    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kserve-container"
Sep 17 16:18:53.976076 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.975951    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f272975-7e98-408c-b98a-7ec39f09044e" containerName="kube-rbac-proxy"
Sep 17 16:18:53.979082 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.979063    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:53.982104 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.982075    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-26f37-predictor-serving-cert\""
Sep 17 16:18:53.982252 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:53.982229    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-26f37-kube-rbac-proxy-sar-config\""
Sep 17 16:18:54.001614 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.001569    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"]
Sep 17 16:18:54.075753 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.075686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c6c613c-70f1-4931-a919-984d068f821a-proxy-tls\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.075873 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.075788    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c6c613c-70f1-4931-a919-984d068f821a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.075873 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.075839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctz2w\" (UniqueName: \"kubernetes.io/projected/5c6c613c-70f1-4931-a919-984d068f821a-kube-api-access-ctz2w\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.177068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.177033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c6c613c-70f1-4931-a919-984d068f821a-proxy-tls\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.177213 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.177087    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c6c613c-70f1-4931-a919-984d068f821a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.177213 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.177112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctz2w\" (UniqueName: \"kubernetes.io/projected/5c6c613c-70f1-4931-a919-984d068f821a-kube-api-access-ctz2w\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.177814 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.177792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c6c613c-70f1-4931-a919-984d068f821a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.179406 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.179384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c6c613c-70f1-4931-a919-984d068f821a-proxy-tls\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.189489 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.189469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctz2w\" (UniqueName: \"kubernetes.io/projected/5c6c613c-70f1-4931-a919-984d068f821a-kube-api-access-ctz2w\") pod \"success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") " pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.291139 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.291119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.566095 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.566072    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"]
Sep 17 16:18:54.567776 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:18:54.567747    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6c613c_70f1_4931_a919_984d068f821a.slice/crio-12f28ce9279599a5822aa6faf80a7087ee52a8a133563efb619d8e04a78a625e WatchSource:0}: Error finding container 12f28ce9279599a5822aa6faf80a7087ee52a8a133563efb619d8e04a78a625e: Status 404 returned error can't find the container with id 12f28ce9279599a5822aa6faf80a7087ee52a8a133563efb619d8e04a78a625e
Sep 17 16:18:54.821198 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.821167    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerID="146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88" exitCode=2
Sep 17 16:18:54.821360 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.821240    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" event={"ID":"5f0ccf71-53ce-4105-a9bc-67974be20b8d","Type":"ContainerDied","Data":"146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88"}
Sep 17 16:18:54.822983 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.822964    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" event={"ID":"5c6c613c-70f1-4931-a919-984d068f821a","Type":"ContainerStarted","Data":"03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00"}
Sep 17 16:18:54.823068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.823013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" event={"ID":"5c6c613c-70f1-4931-a919-984d068f821a","Type":"ContainerStarted","Data":"a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5"}
Sep 17 16:18:54.823068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.823038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" event={"ID":"5c6c613c-70f1-4931-a919-984d068f821a","Type":"ContainerStarted","Data":"12f28ce9279599a5822aa6faf80a7087ee52a8a133563efb619d8e04a78a625e"}
Sep 17 16:18:54.823150 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.823104    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:54.852343 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:54.852244    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podStartSLOduration=1.852223817 podStartE2EDuration="1.852223817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:54.849134366 +0000 UTC m=+1410.148278867" watchObservedRunningTime="2026-09-17 16:18:54.852223817 +0000 UTC m=+1410.151368297"
Sep 17 16:18:55.827248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:55.827213    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:18:55.828515 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:55.828494    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:18:56.530606 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.530544    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 16:18:56.769819 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.769795    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:18:56.798192 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.798135    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls\") pod \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") "
Sep 17 16:18:56.798302 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.798212    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-rbac-proxy-sar-config\") pod \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") "
Sep 17 16:18:56.798353 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.798318    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56pfg\" (UniqueName: \"kubernetes.io/projected/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-api-access-56pfg\") pod \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\" (UID: \"5f0ccf71-53ce-4105-a9bc-67974be20b8d\") "
Sep 17 16:18:56.798610 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.798568    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f0ccf71-53ce-4105-a9bc-67974be20b8d" (UID: "5f0ccf71-53ce-4105-a9bc-67974be20b8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:56.800468 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.800440    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f0ccf71-53ce-4105-a9bc-67974be20b8d" (UID: "5f0ccf71-53ce-4105-a9bc-67974be20b8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:56.801026 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.801004    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-api-access-56pfg" (OuterVolumeSpecName: "kube-api-access-56pfg") pod "5f0ccf71-53ce-4105-a9bc-67974be20b8d" (UID: "5f0ccf71-53ce-4105-a9bc-67974be20b8d"). InnerVolumeSpecName "kube-api-access-56pfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:56.831972 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.831949    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerID="175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46" exitCode=0
Sep 17 16:18:56.832072 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.831997    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"
Sep 17 16:18:56.832072 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.832031    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" event={"ID":"5f0ccf71-53ce-4105-a9bc-67974be20b8d","Type":"ContainerDied","Data":"175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46"}
Sep 17 16:18:56.832174 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.832095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm" event={"ID":"5f0ccf71-53ce-4105-a9bc-67974be20b8d","Type":"ContainerDied","Data":"5cd575b62d72b837f2658fb1e71b933a31baafe0beaa0e90d6a9daccdbd40712"}
Sep 17 16:18:56.832174 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.832147    2557 scope.go:117] "RemoveContainer" containerID="146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88"
Sep 17 16:18:56.832494 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.832468    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:18:56.840366 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.840351    2557 scope.go:117] "RemoveContainer" containerID="175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46"
Sep 17 16:18:56.847352 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.847331    2557 scope.go:117] "RemoveContainer" containerID="146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88"
Sep 17 16:18:56.847675 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:18:56.847656    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88\": container with ID starting with 146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88 not found: ID does not exist" containerID="146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88"
Sep 17 16:18:56.847763 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.847684    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88"} err="failed to get container status \"146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88\": rpc error: code = NotFound desc = could not find container \"146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88\": container with ID starting with 146c2c78ab335aacce7453295ae2d9311e681e152c38afcae7b64eeaa626af88 not found: ID does not exist"
Sep 17 16:18:56.847763 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.847706    2557 scope.go:117] "RemoveContainer" containerID="175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46"
Sep 17 16:18:56.847997 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:18:56.847979    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46\": container with ID starting with 175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46 not found: ID does not exist" containerID="175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46"
Sep 17 16:18:56.848039 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.848003    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46"} err="failed to get container status \"175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46\": rpc error: code = NotFound desc = could not find container \"175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46\": container with ID starting with 175957c08285b710fd110929a1dac13c6367ad952c9b774f1bcbb6c149655a46 not found: ID does not exist"
Sep 17 16:18:56.855328 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.855308    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"]
Sep 17 16:18:56.863572 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.863551    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-48574-predictor-69dd69f964-fv7vm"]
Sep 17 16:18:56.899499 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.899474    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56pfg\" (UniqueName: \"kubernetes.io/projected/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-api-access-56pfg\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:18:56.899614 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.899499    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f0ccf71-53ce-4105-a9bc-67974be20b8d-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:18:56.899614 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:56.899514    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f0ccf71-53ce-4105-a9bc-67974be20b8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:18:57.026965 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:18:57.026936    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" path="/var/lib/kubelet/pods/5f0ccf71-53ce-4105-a9bc-67974be20b8d/volumes"
Sep 17 16:19:01.672632 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:01.672583    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:19:01.836641 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:01.836614    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:19:01.837075 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:01.837044    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:19:11.837813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:11.837725    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:19:21.837108 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:21.837068    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:19:31.837420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:31.837379    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:19:41.837785 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:19:41.837756    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:20:27.016547 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:20:27.016441    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:20:27.019264 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:20:27.019241    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:20:27.021146 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:20:27.021124    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:20:27.023852 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:20:27.023835    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:25:27.046977 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:25:27.046862    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:25:27.051084 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:25:27.051060    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:25:27.052760 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:25:27.052738    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:25:27.055534 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:25:27.055518    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:27:28.546786 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.546755    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"]
Sep 17 16:27:28.547290 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.546967    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container" containerID="cri-o://900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30" gracePeriod=30
Sep 17 16:27:28.547290 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.547009    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kube-rbac-proxy" containerID="cri-o://06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765" gracePeriod=30
Sep 17 16:27:28.701178 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701146    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"]
Sep 17 16:27:28.701514 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701497    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container"
Sep 17 16:27:28.701514 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701513    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container"
Sep 17 16:27:28.701691 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701545    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kube-rbac-proxy"
Sep 17 16:27:28.701691 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701552    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kube-rbac-proxy"
Sep 17 16:27:28.701691 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701628    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kube-rbac-proxy"
Sep 17 16:27:28.701691 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.701637    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0ccf71-53ce-4105-a9bc-67974be20b8d" containerName="kserve-container"
Sep 17 16:27:28.704554 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.704535    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.712713 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.712695    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7ce9a-predictor-serving-cert\""
Sep 17 16:27:28.712985 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.712967    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7ce9a-kube-rbac-proxy-sar-config\""
Sep 17 16:27:28.731840 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.731818    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"]
Sep 17 16:27:28.804660 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.804585    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rs72\" (UniqueName: \"kubernetes.io/projected/440c5089-6096-468c-9246-dafd4c5aae11-kube-api-access-4rs72\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.804660 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.804640    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.804798 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.804667    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440c5089-6096-468c-9246-dafd4c5aae11-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.852740 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.852711    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerID="06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765" exitCode=2
Sep 17 16:27:28.852880 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.852781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" event={"ID":"9b8f143e-2616-4647-8996-2b5e7b4f03c1","Type":"ContainerDied","Data":"06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765"}
Sep 17 16:27:28.905393 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.905366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rs72\" (UniqueName: \"kubernetes.io/projected/440c5089-6096-468c-9246-dafd4c5aae11-kube-api-access-4rs72\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.905513 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.905406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.905513 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.905426    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440c5089-6096-468c-9246-dafd4c5aae11-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.905637 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:27:28.905525    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-serving-cert: secret "success-200-isvc-7ce9a-predictor-serving-cert" not found
Sep 17 16:27:28.905637 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:27:28.905588    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls podName:440c5089-6096-468c-9246-dafd4c5aae11 nodeName:}" failed. No retries permitted until 2026-09-17 16:27:29.405569715 +0000 UTC m=+1924.704714176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls") pod "success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" (UID: "440c5089-6096-468c-9246-dafd4c5aae11") : secret "success-200-isvc-7ce9a-predictor-serving-cert" not found
Sep 17 16:27:28.906111 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.906090    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440c5089-6096-468c-9246-dafd4c5aae11-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:28.922705 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:28.922679    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rs72\" (UniqueName: \"kubernetes.io/projected/440c5089-6096-468c-9246-dafd4c5aae11-kube-api-access-4rs72\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:29.411229 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:29.411197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:29.413461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:29.413437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls\") pod \"success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") " pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:29.613916 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:29.613881    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:29.894215 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:29.894173    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"]
Sep 17 16:27:29.896918 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:27:29.896885    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440c5089_6096_468c_9246_dafd4c5aae11.slice/crio-989ac77d11d38c8f24a64dcc2dfc3e09cd5937602bfb0754d5c2051d3cda405c WatchSource:0}: Error finding container 989ac77d11d38c8f24a64dcc2dfc3e09cd5937602bfb0754d5c2051d3cda405c: Status 404 returned error can't find the container with id 989ac77d11d38c8f24a64dcc2dfc3e09cd5937602bfb0754d5c2051d3cda405c
Sep 17 16:27:30.863281 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:30.863244    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" event={"ID":"440c5089-6096-468c-9246-dafd4c5aae11","Type":"ContainerStarted","Data":"d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485"}
Sep 17 16:27:30.863667 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:30.863316    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:30.863667 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:30.863345    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" event={"ID":"440c5089-6096-468c-9246-dafd4c5aae11","Type":"ContainerStarted","Data":"d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a"}
Sep 17 16:27:30.863667 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:30.863371    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" event={"ID":"440c5089-6096-468c-9246-dafd4c5aae11","Type":"ContainerStarted","Data":"989ac77d11d38c8f24a64dcc2dfc3e09cd5937602bfb0754d5c2051d3cda405c"}
Sep 17 16:27:30.889668 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:30.889621    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podStartSLOduration=2.889606579 podStartE2EDuration="2.889606579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:27:30.887572932 +0000 UTC m=+1926.186717409" watchObservedRunningTime="2026-09-17 16:27:30.889606579 +0000 UTC m=+1926.188751048"
Sep 17 16:27:31.595834 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.595802    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:27:31.631892 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.631853    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-rbac-proxy-sar-config\") pod \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") "
Sep 17 16:27:31.632080 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.631906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzg7h\" (UniqueName: \"kubernetes.io/projected/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-api-access-jzg7h\") pod \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") "
Sep 17 16:27:31.632080 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.631962    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8f143e-2616-4647-8996-2b5e7b4f03c1-proxy-tls\") pod \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\" (UID: \"9b8f143e-2616-4647-8996-2b5e7b4f03c1\") "
Sep 17 16:27:31.632237 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.632210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b8f143e-2616-4647-8996-2b5e7b4f03c1" (UID: "9b8f143e-2616-4647-8996-2b5e7b4f03c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:31.634358 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.634302    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8f143e-2616-4647-8996-2b5e7b4f03c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b8f143e-2616-4647-8996-2b5e7b4f03c1" (UID: "9b8f143e-2616-4647-8996-2b5e7b4f03c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:31.634846 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.634819    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-api-access-jzg7h" (OuterVolumeSpecName: "kube-api-access-jzg7h") pod "9b8f143e-2616-4647-8996-2b5e7b4f03c1" (UID: "9b8f143e-2616-4647-8996-2b5e7b4f03c1"). InnerVolumeSpecName "kube-api-access-jzg7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:31.732828 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.732746    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:27:31.732828 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.732773    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzg7h\" (UniqueName: \"kubernetes.io/projected/9b8f143e-2616-4647-8996-2b5e7b4f03c1-kube-api-access-jzg7h\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:27:31.732828 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.732782    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8f143e-2616-4647-8996-2b5e7b4f03c1-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:27:31.869681 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.869644    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerID="900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30" exitCode=0
Sep 17 16:27:31.870044 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.869701    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"
Sep 17 16:27:31.870044 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.869734    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" event={"ID":"9b8f143e-2616-4647-8996-2b5e7b4f03c1","Type":"ContainerDied","Data":"900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30"}
Sep 17 16:27:31.870044 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.869791    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9" event={"ID":"9b8f143e-2616-4647-8996-2b5e7b4f03c1","Type":"ContainerDied","Data":"ff0a5a1c9537c8846c3f158b767591373bbc66df5b304a6a62413adad04d8a68"}
Sep 17 16:27:31.870044 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.869819    2557 scope.go:117] "RemoveContainer" containerID="06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765"
Sep 17 16:27:31.870331 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.870307    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:31.871722 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.871692    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:27:31.879531 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.879506    2557 scope.go:117] "RemoveContainer" containerID="900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30"
Sep 17 16:27:31.887352 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.887335    2557 scope.go:117] "RemoveContainer" containerID="06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765"
Sep 17 16:27:31.887625 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:27:31.887584    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765\": container with ID starting with 06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765 not found: ID does not exist" containerID="06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765"
Sep 17 16:27:31.887712 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.887630    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765"} err="failed to get container status \"06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765\": rpc error: code = NotFound desc = could not find container \"06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765\": container with ID starting with 06280cbdf36e7ad3ba4103c3fc3ea47157d59d9346ac833e3472272323054765 not found: ID does not exist"
Sep 17 16:27:31.887712 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.887646    2557 scope.go:117] "RemoveContainer" containerID="900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30"
Sep 17 16:27:31.887887 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:27:31.887868    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30\": container with ID starting with 900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30 not found: ID does not exist" containerID="900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30"
Sep 17 16:27:31.887919 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.887894    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30"} err="failed to get container status \"900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30\": rpc error: code = NotFound desc = could not find container \"900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30\": container with ID starting with 900f53634639cc1ad7df85213f9d05cd39c4ede2accce6e7430cd26978896b30 not found: ID does not exist"
Sep 17 16:27:31.900325 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.900303    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"]
Sep 17 16:27:31.903705 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:31.903684    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e5cb-predictor-5d6b764d8-286f9"]
Sep 17 16:27:32.874529 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:32.874493    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:27:33.027218 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:33.027179    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" path="/var/lib/kubelet/pods/9b8f143e-2616-4647-8996-2b5e7b4f03c1/volumes"
Sep 17 16:27:37.878862 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:37.878831    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:27:37.879422 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:37.879259    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:27:47.879721 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:47.879687    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:27:57.880266 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:27:57.880223    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:28:07.879720 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:07.879679    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:28:08.671281 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.671245    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"]
Sep 17 16:28:08.671536 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.671491    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container" containerID="cri-o://a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5" gracePeriod=30
Sep 17 16:28:08.671651 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.671574    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kube-rbac-proxy" containerID="cri-o://03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00" gracePeriod=30
Sep 17 16:28:08.774072 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774035    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"]
Sep 17 16:28:08.774365 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774353    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kube-rbac-proxy"
Sep 17 16:28:08.774413 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774366    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kube-rbac-proxy"
Sep 17 16:28:08.774413 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774387    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container"
Sep 17 16:28:08.774413 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774393    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container"
Sep 17 16:28:08.774520 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774445    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kube-rbac-proxy"
Sep 17 16:28:08.774520 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.774453    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8f143e-2616-4647-8996-2b5e7b4f03c1" containerName="kserve-container"
Sep 17 16:28:08.777611 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.777564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:08.784431 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.784408    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-37c70-predictor-serving-cert\""
Sep 17 16:28:08.784536 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.784466    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-37c70-kube-rbac-proxy-sar-config\""
Sep 17 16:28:08.797025 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.797004    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"]
Sep 17 16:28:08.928505 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.928427    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:08.928860 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.928515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-proxy-tls\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:08.928860 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:08.928548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrsh\" (UniqueName: \"kubernetes.io/projected/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-api-access-nwrsh\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.021984 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.021940    2557 generic.go:356] "Generic (PLEG): container finished" podID="5c6c613c-70f1-4931-a919-984d068f821a" containerID="03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00" exitCode=2
Sep 17 16:28:09.031225 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.027959    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" event={"ID":"5c6c613c-70f1-4931-a919-984d068f821a","Type":"ContainerDied","Data":"03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00"}
Sep 17 16:28:09.031844 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.031813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-proxy-tls\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.031946 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.031856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrsh\" (UniqueName: \"kubernetes.io/projected/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-api-access-nwrsh\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.031946 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.031873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.032467 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.032450    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.034151 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.034129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-proxy-tls\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.041462 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.041444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrsh\" (UniqueName: \"kubernetes.io/projected/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-api-access-nwrsh\") pod \"success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") " pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.087273 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.087246    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:09.360848 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:09.360823    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"]
Sep 17 16:28:09.363253 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:28:09.363231    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cf25ad_3c17_44f1_b5ae_b2a0b621d1ef.slice/crio-82794bace7e7ac516c69435b44e45028c8e81d1067040af1e46b192d3e5bf2ad WatchSource:0}: Error finding container 82794bace7e7ac516c69435b44e45028c8e81d1067040af1e46b192d3e5bf2ad: Status 404 returned error can't find the container with id 82794bace7e7ac516c69435b44e45028c8e81d1067040af1e46b192d3e5bf2ad
Sep 17 16:28:10.027922 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.027888    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" event={"ID":"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef","Type":"ContainerStarted","Data":"c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf"}
Sep 17 16:28:10.028381 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.027951    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:10.028381 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.027980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" event={"ID":"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef","Type":"ContainerStarted","Data":"0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd"}
Sep 17 16:28:10.028381 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.028012    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" event={"ID":"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef","Type":"ContainerStarted","Data":"82794bace7e7ac516c69435b44e45028c8e81d1067040af1e46b192d3e5bf2ad"}
Sep 17 16:28:10.028381 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.028043    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:10.029385 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.029358    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:28:10.050420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:10.050354    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podStartSLOduration=2.050333968 podStartE2EDuration="2.050333968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:10.047910879 +0000 UTC m=+1965.347055357" watchObservedRunningTime="2026-09-17 16:28:10.050333968 +0000 UTC m=+1965.349478445"
Sep 17 16:28:11.031765 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.031724    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:28:11.711683 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.711660    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:28:11.855654 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.855573    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c6c613c-70f1-4931-a919-984d068f821a-kube-rbac-proxy-sar-config\") pod \"5c6c613c-70f1-4931-a919-984d068f821a\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") "
Sep 17 16:28:11.855654 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.855625    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c6c613c-70f1-4931-a919-984d068f821a-proxy-tls\") pod \"5c6c613c-70f1-4931-a919-984d068f821a\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") "
Sep 17 16:28:11.855868 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.855757    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctz2w\" (UniqueName: \"kubernetes.io/projected/5c6c613c-70f1-4931-a919-984d068f821a-kube-api-access-ctz2w\") pod \"5c6c613c-70f1-4931-a919-984d068f821a\" (UID: \"5c6c613c-70f1-4931-a919-984d068f821a\") "
Sep 17 16:28:11.855936 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.855915    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6c613c-70f1-4931-a919-984d068f821a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c6c613c-70f1-4931-a919-984d068f821a" (UID: "5c6c613c-70f1-4931-a919-984d068f821a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:11.856033 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.856019    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c6c613c-70f1-4931-a919-984d068f821a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:28:11.857552 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.857529    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6c613c-70f1-4931-a919-984d068f821a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c6c613c-70f1-4931-a919-984d068f821a" (UID: "5c6c613c-70f1-4931-a919-984d068f821a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:11.857696 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.857652    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6c613c-70f1-4931-a919-984d068f821a-kube-api-access-ctz2w" (OuterVolumeSpecName: "kube-api-access-ctz2w") pod "5c6c613c-70f1-4931-a919-984d068f821a" (UID: "5c6c613c-70f1-4931-a919-984d068f821a"). InnerVolumeSpecName "kube-api-access-ctz2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:11.956759 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.956736    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c6c613c-70f1-4931-a919-984d068f821a-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:28:11.956759 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:11.956758    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctz2w\" (UniqueName: \"kubernetes.io/projected/5c6c613c-70f1-4931-a919-984d068f821a-kube-api-access-ctz2w\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:28:12.037387 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.037362    2557 generic.go:356] "Generic (PLEG): container finished" podID="5c6c613c-70f1-4931-a919-984d068f821a" containerID="a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5" exitCode=0
Sep 17 16:28:12.037728 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.037405    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"
Sep 17 16:28:12.037728 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.037434    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" event={"ID":"5c6c613c-70f1-4931-a919-984d068f821a","Type":"ContainerDied","Data":"a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5"}
Sep 17 16:28:12.037728 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.037478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt" event={"ID":"5c6c613c-70f1-4931-a919-984d068f821a","Type":"ContainerDied","Data":"12f28ce9279599a5822aa6faf80a7087ee52a8a133563efb619d8e04a78a625e"}
Sep 17 16:28:12.037728 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.037503    2557 scope.go:117] "RemoveContainer" containerID="03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00"
Sep 17 16:28:12.045809 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.045794    2557 scope.go:117] "RemoveContainer" containerID="a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5"
Sep 17 16:28:12.054301 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.054279    2557 scope.go:117] "RemoveContainer" containerID="03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00"
Sep 17 16:28:12.054542 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:28:12.054525    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00\": container with ID starting with 03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00 not found: ID does not exist" containerID="03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00"
Sep 17 16:28:12.054603 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.054549    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00"} err="failed to get container status \"03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00\": rpc error: code = NotFound desc = could not find container \"03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00\": container with ID starting with 03bd8fe228413d57289ecbb0413b3c2dd2b0c0e8b88325fda3de78ac66c77f00 not found: ID does not exist"
Sep 17 16:28:12.054603 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.054563    2557 scope.go:117] "RemoveContainer" containerID="a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5"
Sep 17 16:28:12.054783 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:28:12.054769    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5\": container with ID starting with a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5 not found: ID does not exist" containerID="a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5"
Sep 17 16:28:12.054821 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.054787    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5"} err="failed to get container status \"a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5\": rpc error: code = NotFound desc = could not find container \"a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5\": container with ID starting with a859fe4e7e7270256b8f14d00853c9d422bf0962610bfd69a231e65d8c4871b5 not found: ID does not exist"
Sep 17 16:28:12.061183 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.061161    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"]
Sep 17 16:28:12.067824 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:12.067805    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26f37-predictor-7c64bddcdd-cg5kt"]
Sep 17 16:28:13.027866 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:13.027833    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6c613c-70f1-4931-a919-984d068f821a" path="/var/lib/kubelet/pods/5c6c613c-70f1-4931-a919-984d068f821a/volumes"
Sep 17 16:28:16.036111 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:16.036080    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:16.036529 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:16.036470    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:28:17.879738 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:17.879712    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:28:26.037426 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:26.037384    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:28:36.036607 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:36.036539    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:28:38.937648 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:38.937611    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"]
Sep 17 16:28:38.938098 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:38.937893    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container" containerID="cri-o://d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a" gracePeriod=30
Sep 17 16:28:38.938098 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:38.937931    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kube-rbac-proxy" containerID="cri-o://d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485" gracePeriod=30
Sep 17 16:28:39.145938 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.145901    2557 generic.go:356] "Generic (PLEG): container finished" podID="440c5089-6096-468c-9246-dafd4c5aae11" containerID="d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485" exitCode=2
Sep 17 16:28:39.146111 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.145960    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" event={"ID":"440c5089-6096-468c-9246-dafd4c5aae11","Type":"ContainerDied","Data":"d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485"}
Sep 17 16:28:39.149337 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149307    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"]
Sep 17 16:28:39.149716 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149701    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container"
Sep 17 16:28:39.149810 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149718    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container"
Sep 17 16:28:39.149810 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149762    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kube-rbac-proxy"
Sep 17 16:28:39.149810 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149772    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kube-rbac-proxy"
Sep 17 16:28:39.149967 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149848    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kube-rbac-proxy"
Sep 17 16:28:39.149967 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.149866    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6c613c-70f1-4931-a919-984d068f821a" containerName="kserve-container"
Sep 17 16:28:39.152911 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.152893    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.155378 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.155356    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6d5c6-predictor-serving-cert\""
Sep 17 16:28:39.155474 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.155379    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6d5c6-kube-rbac-proxy-sar-config\""
Sep 17 16:28:39.169555 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.169524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"]
Sep 17 16:28:39.284298 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.284209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.284298 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.284271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8t4p\" (UniqueName: \"kubernetes.io/projected/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-api-access-d8t4p\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.284489 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.284354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.384928 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.384889    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.385118 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.384956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.385118 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.385001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t4p\" (UniqueName: \"kubernetes.io/projected/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-api-access-d8t4p\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.385118 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:28:39.385029    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-serving-cert: secret "success-200-isvc-6d5c6-predictor-serving-cert" not found
Sep 17 16:28:39.385118 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:28:39.385099    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls podName:bff3aff9-526f-4fe0-aaf7-a45a1057942e nodeName:}" failed. No retries permitted until 2026-09-17 16:28:39.885084681 +0000 UTC m=+1995.184229150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls") pod "success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" (UID: "bff3aff9-526f-4fe0-aaf7-a45a1057942e") : secret "success-200-isvc-6d5c6-predictor-serving-cert" not found
Sep 17 16:28:39.385576 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.385558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.395962 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.395937    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8t4p\" (UniqueName: \"kubernetes.io/projected/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-api-access-d8t4p\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.889248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.889212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:39.891510 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:39.891484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls\") pod \"success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") " pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:40.063094 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:40.063053    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:40.334800 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:40.334775    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"]
Sep 17 16:28:40.336975 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:28:40.336945    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff3aff9_526f_4fe0_aaf7_a45a1057942e.slice/crio-ad35f48bc6cbe21d6cc36bb2211e66171f900dd3b5bc2336bd21f4e615a48e9b WatchSource:0}: Error finding container ad35f48bc6cbe21d6cc36bb2211e66171f900dd3b5bc2336bd21f4e615a48e9b: Status 404 returned error can't find the container with id ad35f48bc6cbe21d6cc36bb2211e66171f900dd3b5bc2336bd21f4e615a48e9b
Sep 17 16:28:41.156787 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:41.156743    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" event={"ID":"bff3aff9-526f-4fe0-aaf7-a45a1057942e","Type":"ContainerStarted","Data":"9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d"}
Sep 17 16:28:41.157188 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:41.156817    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" event={"ID":"bff3aff9-526f-4fe0-aaf7-a45a1057942e","Type":"ContainerStarted","Data":"0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86"}
Sep 17 16:28:41.157188 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:41.156854    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:41.157188 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:41.156886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" event={"ID":"bff3aff9-526f-4fe0-aaf7-a45a1057942e","Type":"ContainerStarted","Data":"ad35f48bc6cbe21d6cc36bb2211e66171f900dd3b5bc2336bd21f4e615a48e9b"}
Sep 17 16:28:41.184136 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:41.184090    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podStartSLOduration=2.184075119 podStartE2EDuration="2.184075119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:41.18357006 +0000 UTC m=+1996.482714536" watchObservedRunningTime="2026-09-17 16:28:41.184075119 +0000 UTC m=+1996.483219596"
Sep 17 16:28:42.078655 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.078628    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:28:42.162534 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.162503    2557 generic.go:356] "Generic (PLEG): container finished" podID="440c5089-6096-468c-9246-dafd4c5aae11" containerID="d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a" exitCode=0
Sep 17 16:28:42.162970 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.162548    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"
Sep 17 16:28:42.162970 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.162568    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" event={"ID":"440c5089-6096-468c-9246-dafd4c5aae11","Type":"ContainerDied","Data":"d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a"}
Sep 17 16:28:42.162970 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.162667    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j" event={"ID":"440c5089-6096-468c-9246-dafd4c5aae11","Type":"ContainerDied","Data":"989ac77d11d38c8f24a64dcc2dfc3e09cd5937602bfb0754d5c2051d3cda405c"}
Sep 17 16:28:42.162970 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.162710    2557 scope.go:117] "RemoveContainer" containerID="d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485"
Sep 17 16:28:42.162970 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.162955    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:42.164071 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.164048    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:28:42.170572 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.170556    2557 scope.go:117] "RemoveContainer" containerID="d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a"
Sep 17 16:28:42.177286 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.177269    2557 scope.go:117] "RemoveContainer" containerID="d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485"
Sep 17 16:28:42.177511 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:28:42.177493    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485\": container with ID starting with d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485 not found: ID does not exist" containerID="d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485"
Sep 17 16:28:42.177564 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.177519    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485"} err="failed to get container status \"d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485\": rpc error: code = NotFound desc = could not find container \"d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485\": container with ID starting with d371bf372dfc86bfb31aeb7f08cfed38fc89b08ba7df2616af8b8020d66af485 not found: ID does not exist"
Sep 17 16:28:42.177564 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.177539    2557 scope.go:117] "RemoveContainer" containerID="d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a"
Sep 17 16:28:42.177803 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:28:42.177788    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a\": container with ID starting with d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a not found: ID does not exist" containerID="d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a"
Sep 17 16:28:42.177847 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.177807    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a"} err="failed to get container status \"d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a\": rpc error: code = NotFound desc = could not find container \"d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a\": container with ID starting with d040c8280c7a8ef3a8987654170fbc8a7dfbedce9c0adb161f4c8be9d294813a not found: ID does not exist"
Sep 17 16:28:42.210069 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.210050    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls\") pod \"440c5089-6096-468c-9246-dafd4c5aae11\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") "
Sep 17 16:28:42.210138 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.210084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rs72\" (UniqueName: \"kubernetes.io/projected/440c5089-6096-468c-9246-dafd4c5aae11-kube-api-access-4rs72\") pod \"440c5089-6096-468c-9246-dafd4c5aae11\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") "
Sep 17 16:28:42.210280 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.210264    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440c5089-6096-468c-9246-dafd4c5aae11-kube-rbac-proxy-sar-config\") pod \"440c5089-6096-468c-9246-dafd4c5aae11\" (UID: \"440c5089-6096-468c-9246-dafd4c5aae11\") "
Sep 17 16:28:42.210579 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.210561    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440c5089-6096-468c-9246-dafd4c5aae11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "440c5089-6096-468c-9246-dafd4c5aae11" (UID: "440c5089-6096-468c-9246-dafd4c5aae11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:42.211860 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.211839    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440c5089-6096-468c-9246-dafd4c5aae11-kube-api-access-4rs72" (OuterVolumeSpecName: "kube-api-access-4rs72") pod "440c5089-6096-468c-9246-dafd4c5aae11" (UID: "440c5089-6096-468c-9246-dafd4c5aae11"). InnerVolumeSpecName "kube-api-access-4rs72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:42.212092 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.212075    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "440c5089-6096-468c-9246-dafd4c5aae11" (UID: "440c5089-6096-468c-9246-dafd4c5aae11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:42.311813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.311788    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440c5089-6096-468c-9246-dafd4c5aae11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:28:42.311813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.311807    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440c5089-6096-468c-9246-dafd4c5aae11-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:28:42.311813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.311820    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rs72\" (UniqueName: \"kubernetes.io/projected/440c5089-6096-468c-9246-dafd4c5aae11-kube-api-access-4rs72\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:28:42.503426 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.503396    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"]
Sep 17 16:28:42.509577 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:42.509554    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ce9a-predictor-85fc555b5d-zzg4j"]
Sep 17 16:28:43.027464 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:43.027432    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440c5089-6096-468c-9246-dafd4c5aae11" path="/var/lib/kubelet/pods/440c5089-6096-468c-9246-dafd4c5aae11/volumes"
Sep 17 16:28:43.168098 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:43.168064    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:28:46.036779 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:46.036738    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:28:48.172284 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:48.172248    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:28:48.172740 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:48.172710    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:28:56.037481 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:56.037444    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:28:58.172748 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:28:58.172705    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:29:08.172933 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:29:08.172889    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:29:18.173489 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:29:18.173448    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:29:28.173577 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:29:28.173548    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:30:27.075182 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:30:27.075081    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:30:27.079861 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:30:27.079845    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:30:27.079958 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:30:27.079845    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:30:27.084320 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:30:27.084305    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:35:27.102499 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:35:27.102400    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:35:27.107087 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:35:27.107066    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:35:27.108010 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:35:27.107995    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:35:27.112549 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:35:27.112536    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:37:54.056280 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:54.056248    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"]
Sep 17 16:37:54.056811 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:54.056446    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container" containerID="cri-o://0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86" gracePeriod=30
Sep 17 16:37:54.056811 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:54.056487    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kube-rbac-proxy" containerID="cri-o://9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d" gracePeriod=30
Sep 17 16:37:54.403645 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:54.403544    2557 generic.go:356] "Generic (PLEG): container finished" podID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerID="9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d" exitCode=2
Sep 17 16:37:54.403645 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:54.403619    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" event={"ID":"bff3aff9-526f-4fe0-aaf7-a45a1057942e","Type":"ContainerDied","Data":"9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d"}
Sep 17 16:37:57.191746 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.191719    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:37:57.332654 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.332569    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls\") pod \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") "
Sep 17 16:37:57.332654 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.332648    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-rbac-proxy-sar-config\") pod \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") "
Sep 17 16:37:57.332874 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.332721    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8t4p\" (UniqueName: \"kubernetes.io/projected/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-api-access-d8t4p\") pod \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\" (UID: \"bff3aff9-526f-4fe0-aaf7-a45a1057942e\") "
Sep 17 16:37:57.332996 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.332971    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bff3aff9-526f-4fe0-aaf7-a45a1057942e" (UID: "bff3aff9-526f-4fe0-aaf7-a45a1057942e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:37:57.334813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.334773    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-api-access-d8t4p" (OuterVolumeSpecName: "kube-api-access-d8t4p") pod "bff3aff9-526f-4fe0-aaf7-a45a1057942e" (UID: "bff3aff9-526f-4fe0-aaf7-a45a1057942e"). InnerVolumeSpecName "kube-api-access-d8t4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:37:57.334813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.334802    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bff3aff9-526f-4fe0-aaf7-a45a1057942e" (UID: "bff3aff9-526f-4fe0-aaf7-a45a1057942e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:37:57.417299 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.417267    2557 generic.go:356] "Generic (PLEG): container finished" podID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerID="0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86" exitCode=0
Sep 17 16:37:57.417461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.417320    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"
Sep 17 16:37:57.417461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.417344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" event={"ID":"bff3aff9-526f-4fe0-aaf7-a45a1057942e","Type":"ContainerDied","Data":"0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86"}
Sep 17 16:37:57.417461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.417395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv" event={"ID":"bff3aff9-526f-4fe0-aaf7-a45a1057942e","Type":"ContainerDied","Data":"ad35f48bc6cbe21d6cc36bb2211e66171f900dd3b5bc2336bd21f4e615a48e9b"}
Sep 17 16:37:57.417461 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.417421    2557 scope.go:117] "RemoveContainer" containerID="9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d"
Sep 17 16:37:57.426071 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.426045    2557 scope.go:117] "RemoveContainer" containerID="0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86"
Sep 17 16:37:57.433162 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433146    2557 scope.go:117] "RemoveContainer" containerID="9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d"
Sep 17 16:37:57.433469 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:37:57.433449    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d\": container with ID starting with 9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d not found: ID does not exist" containerID="9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d"
Sep 17 16:37:57.433517 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433479    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d"} err="failed to get container status \"9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d\": rpc error: code = NotFound desc = could not find container \"9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d\": container with ID starting with 9919da5bb58aeafd65c9d6ef0e7b889aa9e4410a639b1acfb498a9910d9bc21d not found: ID does not exist"
Sep 17 16:37:57.433517 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433498    2557 scope.go:117] "RemoveContainer" containerID="0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86"
Sep 17 16:37:57.433624 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433523    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:37:57.433624 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433542    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8t4p\" (UniqueName: \"kubernetes.io/projected/bff3aff9-526f-4fe0-aaf7-a45a1057942e-kube-api-access-d8t4p\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:37:57.433624 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433551    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bff3aff9-526f-4fe0-aaf7-a45a1057942e-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:37:57.433763 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:37:57.433743    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86\": container with ID starting with 0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86 not found: ID does not exist" containerID="0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86"
Sep 17 16:37:57.433810 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.433769    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86"} err="failed to get container status \"0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86\": rpc error: code = NotFound desc = could not find container \"0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86\": container with ID starting with 0388654442d094809122aaeef87433dfa9437f96f788cf6583d413a133141f86 not found: ID does not exist"
Sep 17 16:37:57.451622 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.451570    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"]
Sep 17 16:37:57.464575 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:57.464552    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d5c6-predictor-b6584cc56-qrgxv"]
Sep 17 16:37:59.026969 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:37:59.026939    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" path="/var/lib/kubelet/pods/bff3aff9-526f-4fe0-aaf7-a45a1057942e/volumes"
Sep 17 16:40:27.134746 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:40:27.134642    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:40:27.139659 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:40:27.139644    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:40:27.140437 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:40:27.140410    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:40:27.144799 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:40:27.144784    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:45:27.161069 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:27.161042    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:45:27.165925 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:27.165894    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:45:27.167666 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:27.167639    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:45:27.172002 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:27.171984    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:45:28.386418 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:28.386380    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"]
Sep 17 16:45:28.386881 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:28.386679    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container" containerID="cri-o://0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd" gracePeriod=30
Sep 17 16:45:28.386881 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:28.386753    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kube-rbac-proxy" containerID="cri-o://c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf" gracePeriod=30
Sep 17 16:45:29.177724 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.177684    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerID="c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf" exitCode=2
Sep 17 16:45:29.177886 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.177754    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" event={"ID":"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef","Type":"ContainerDied","Data":"c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf"}
Sep 17 16:45:29.947820 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.947787    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tj7nf/must-gather-8cwfd"]
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948095    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kube-rbac-proxy"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948105    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kube-rbac-proxy"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948115    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kube-rbac-proxy"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948122    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kube-rbac-proxy"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948139    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948145    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948155    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948161    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948212    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kserve-container"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948218    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="440c5089-6096-468c-9246-dafd4c5aae11" containerName="kube-rbac-proxy"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948223    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kserve-container"
Sep 17 16:45:29.948260 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.948231    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff3aff9-526f-4fe0-aaf7-a45a1057942e" containerName="kube-rbac-proxy"
Sep 17 16:45:29.951344 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.951320    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:29.954435 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.954417    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tj7nf\"/\"default-dockercfg-vwbnk\""
Sep 17 16:45:29.955164 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.955147    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tj7nf\"/\"kube-root-ca.crt\""
Sep 17 16:45:29.955268 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.955181    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tj7nf\"/\"openshift-service-ca.crt\""
Sep 17 16:45:29.964887 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:29.964867    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tj7nf/must-gather-8cwfd"]
Sep 17 16:45:30.065626 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.065568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-must-gather-output\") pod \"must-gather-8cwfd\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") " pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.065788 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.065673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8sx9\" (UniqueName: \"kubernetes.io/projected/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-kube-api-access-v8sx9\") pod \"must-gather-8cwfd\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") " pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.166900 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.166862    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8sx9\" (UniqueName: \"kubernetes.io/projected/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-kube-api-access-v8sx9\") pod \"must-gather-8cwfd\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") " pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.167092 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.167007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-must-gather-output\") pod \"must-gather-8cwfd\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") " pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.167417 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.167397    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-must-gather-output\") pod \"must-gather-8cwfd\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") " pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.178743 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.178722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8sx9\" (UniqueName: \"kubernetes.io/projected/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-kube-api-access-v8sx9\") pod \"must-gather-8cwfd\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") " pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.279231 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.279150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:45:30.559730 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.559705    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tj7nf/must-gather-8cwfd"]
Sep 17 16:45:30.561765 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:45:30.561736    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcda84fb_cb10_4b2a_941b_1a34a1f3b894.slice/crio-4b9fbfe42befca3f0f18444cb49f5a36c6ea8e13e86df01620914263ac4b01ef WatchSource:0}: Error finding container 4b9fbfe42befca3f0f18444cb49f5a36c6ea8e13e86df01620914263ac4b01ef: Status 404 returned error can't find the container with id 4b9fbfe42befca3f0f18444cb49f5a36c6ea8e13e86df01620914263ac4b01ef
Sep 17 16:45:30.563468 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:30.563453    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:45:31.031978 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.031940    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 16:45:31.186506 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.186468    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tj7nf/must-gather-8cwfd" event={"ID":"bcda84fb-cb10-4b2a-941b-1a34a1f3b894","Type":"ContainerStarted","Data":"4b9fbfe42befca3f0f18444cb49f5a36c6ea8e13e86df01620914263ac4b01ef"}
Sep 17 16:45:31.428813 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.428790    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:45:31.477528 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.477497    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwrsh\" (UniqueName: \"kubernetes.io/projected/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-api-access-nwrsh\") pod \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") "
Sep 17 16:45:31.477712 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.477609    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-rbac-proxy-sar-config\") pod \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") "
Sep 17 16:45:31.477712 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.477668    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-proxy-tls\") pod \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\" (UID: \"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef\") "
Sep 17 16:45:31.477920 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.477885    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" (UID: "a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:31.479836 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.479809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-api-access-nwrsh" (OuterVolumeSpecName: "kube-api-access-nwrsh") pod "a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" (UID: "a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef"). InnerVolumeSpecName "kube-api-access-nwrsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:31.479977 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.479958    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" (UID: "a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:31.578535 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.578465    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:45:31.578535 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.578489    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-proxy-tls\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:45:31.578535 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:31.578499    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrsh\" (UniqueName: \"kubernetes.io/projected/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef-kube-api-access-nwrsh\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:45:32.196420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.195292    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerID="0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd" exitCode=0
Sep 17 16:45:32.196420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.195417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" event={"ID":"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef","Type":"ContainerDied","Data":"0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd"}
Sep 17 16:45:32.196420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.195479    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk" event={"ID":"a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef","Type":"ContainerDied","Data":"82794bace7e7ac516c69435b44e45028c8e81d1067040af1e46b192d3e5bf2ad"}
Sep 17 16:45:32.196420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.195517    2557 scope.go:117] "RemoveContainer" containerID="c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf"
Sep 17 16:45:32.196420 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.195692    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"
Sep 17 16:45:32.212898 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.212717    2557 scope.go:117] "RemoveContainer" containerID="0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd"
Sep 17 16:45:32.228005 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.227951    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"]
Sep 17 16:45:32.230444 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.230343    2557 scope.go:117] "RemoveContainer" containerID="c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf"
Sep 17 16:45:32.231906 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:45:32.230744    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf\": container with ID starting with c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf not found: ID does not exist" containerID="c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf"
Sep 17 16:45:32.231906 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.230778    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf"} err="failed to get container status \"c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf\": rpc error: code = NotFound desc = could not find container \"c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf\": container with ID starting with c56a5564e3f2bd37c367756076eae08476765ffa552955ecf5a0459dfd4cd8bf not found: ID does not exist"
Sep 17 16:45:32.231906 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.230800    2557 scope.go:117] "RemoveContainer" containerID="0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd"
Sep 17 16:45:32.232178 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:45:32.232057    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd\": container with ID starting with 0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd not found: ID does not exist" containerID="0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd"
Sep 17 16:45:32.232178 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.232085    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd"} err="failed to get container status \"0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd\": rpc error: code = NotFound desc = could not find container \"0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd\": container with ID starting with 0375b8f2608700db743a67a37ccebef2c99a871fc16c525e9c0f138f0f7c4fdd not found: ID does not exist"
Sep 17 16:45:32.234721 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:32.234663    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-37c70-predictor-6b44d5b64d-bp6hk"]
Sep 17 16:45:33.038166 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:33.038131    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" path="/var/lib/kubelet/pods/a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef/volumes"
Sep 17 16:45:38.230127 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:38.230086    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tj7nf/must-gather-8cwfd" event={"ID":"bcda84fb-cb10-4b2a-941b-1a34a1f3b894","Type":"ContainerStarted","Data":"2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859"}
Sep 17 16:45:38.230586 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:38.230140    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tj7nf/must-gather-8cwfd" event={"ID":"bcda84fb-cb10-4b2a-941b-1a34a1f3b894","Type":"ContainerStarted","Data":"c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"}
Sep 17 16:45:38.249409 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:38.249342    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tj7nf/must-gather-8cwfd" podStartSLOduration=2.466318539 podStartE2EDuration="9.249321906s" totalImagesPullingTime="6.783003367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:45:38.247903131 +0000 UTC m=+3013.547047609" watchObservedRunningTime="2026-09-17 16:45:38.249321906 +0000 UTC m=+3013.548466375"
Sep 17 16:45:58.319719 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:58.319686    2557 generic.go:356] "Generic (PLEG): container finished" podID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerID="c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714" exitCode=0
Sep 17 16:45:58.320104 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:58.319758    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tj7nf/must-gather-8cwfd" event={"ID":"bcda84fb-cb10-4b2a-941b-1a34a1f3b894","Type":"ContainerDied","Data":"c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"}
Sep 17 16:45:58.320104 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:58.320079    2557 scope.go:117] "RemoveContainer" containerID="c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"
Sep 17 16:45:59.205881 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.205855    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tj7nf_must-gather-8cwfd_bcda84fb-cb10-4b2a-941b-1a34a1f3b894/gather/0.log"
Sep 17 16:45:59.814906 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.814870    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pmzkb/must-gather-br2fm"]
Sep 17 16:45:59.815262 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.815251    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container"
Sep 17 16:45:59.815303 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.815265    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container"
Sep 17 16:45:59.815303 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.815272    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kube-rbac-proxy"
Sep 17 16:45:59.815303 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.815277    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kube-rbac-proxy"
Sep 17 16:45:59.815397 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.815351    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kserve-container"
Sep 17 16:45:59.815397 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.815359    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cf25ad-3c17-44f1-b5ae-b2a0b621d1ef" containerName="kube-rbac-proxy"
Sep 17 16:45:59.817847 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.817823    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:45:59.818091 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.818067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj75d\" (UniqueName: \"kubernetes.io/projected/11893dba-b91e-4eb8-ac9d-7980f5670525-kube-api-access-hj75d\") pod \"must-gather-br2fm\" (UID: \"11893dba-b91e-4eb8-ac9d-7980f5670525\") " pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:45:59.818174 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.818137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11893dba-b91e-4eb8-ac9d-7980f5670525-must-gather-output\") pod \"must-gather-br2fm\" (UID: \"11893dba-b91e-4eb8-ac9d-7980f5670525\") " pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:45:59.820284 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.820266    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmzkb\"/\"openshift-service-ca.crt\""
Sep 17 16:45:59.820341 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.820270    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pmzkb\"/\"default-dockercfg-7gfcx\""
Sep 17 16:45:59.820341 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.820308    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmzkb\"/\"kube-root-ca.crt\""
Sep 17 16:45:59.844418 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.844394    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmzkb/must-gather-br2fm"]
Sep 17 16:45:59.918851 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.918826    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj75d\" (UniqueName: \"kubernetes.io/projected/11893dba-b91e-4eb8-ac9d-7980f5670525-kube-api-access-hj75d\") pod \"must-gather-br2fm\" (UID: \"11893dba-b91e-4eb8-ac9d-7980f5670525\") " pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:45:59.918993 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.918859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11893dba-b91e-4eb8-ac9d-7980f5670525-must-gather-output\") pod \"must-gather-br2fm\" (UID: \"11893dba-b91e-4eb8-ac9d-7980f5670525\") " pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:45:59.919149 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.919133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11893dba-b91e-4eb8-ac9d-7980f5670525-must-gather-output\") pod \"must-gather-br2fm\" (UID: \"11893dba-b91e-4eb8-ac9d-7980f5670525\") " pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:45:59.928662 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:45:59.928637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj75d\" (UniqueName: \"kubernetes.io/projected/11893dba-b91e-4eb8-ac9d-7980f5670525-kube-api-access-hj75d\") pod \"must-gather-br2fm\" (UID: \"11893dba-b91e-4eb8-ac9d-7980f5670525\") " pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:46:00.126954 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:00.126886    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmzkb/must-gather-br2fm"
Sep 17 16:46:00.397949 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:00.397923    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmzkb/must-gather-br2fm"]
Sep 17 16:46:00.400266 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:46:00.400227    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11893dba_b91e_4eb8_ac9d_7980f5670525.slice/crio-798b3499f2cae365d63fbd9c12c6469f47aefbd88dfc23e11f34587ec6ee3576 WatchSource:0}: Error finding container 798b3499f2cae365d63fbd9c12c6469f47aefbd88dfc23e11f34587ec6ee3576: Status 404 returned error can't find the container with id 798b3499f2cae365d63fbd9c12c6469f47aefbd88dfc23e11f34587ec6ee3576
Sep 17 16:46:01.334574 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:01.334535    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/must-gather-br2fm" event={"ID":"11893dba-b91e-4eb8-ac9d-7980f5670525","Type":"ContainerStarted","Data":"798b3499f2cae365d63fbd9c12c6469f47aefbd88dfc23e11f34587ec6ee3576"}
Sep 17 16:46:02.340659 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:02.340628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/must-gather-br2fm" event={"ID":"11893dba-b91e-4eb8-ac9d-7980f5670525","Type":"ContainerStarted","Data":"71336a5fd7c8ba4764c4ee0b565fb328289f850e956f3389ed6edbcd2ce8204b"}
Sep 17 16:46:02.341105 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:02.340670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/must-gather-br2fm" event={"ID":"11893dba-b91e-4eb8-ac9d-7980f5670525","Type":"ContainerStarted","Data":"b9790a7d303f268e03d5c9bf74dedf1f29800409375ada89652aee8338bca91f"}
Sep 17 16:46:02.357281 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:02.357235    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pmzkb/must-gather-br2fm" podStartSLOduration=2.132934265 podStartE2EDuration="3.35721924s" totalImagesPullingTime="1.224284975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:46:02.356679599 +0000 UTC m=+3037.655824078" watchObservedRunningTime="2026-09-17 16:46:02.35721924 +0000 UTC m=+3037.656363788"
Sep 17 16:46:03.061315 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:03.061279    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vqfpz_d18cce37-76b4-406d-b113-b47a3c850d05/global-pull-secret-syncer/0.log"
Sep 17 16:46:03.257582 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:03.257549    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wvx8p_5829edb8-7ad1-4a80-978f-29f5f80d8140/konnectivity-agent/0.log"
Sep 17 16:46:03.321810 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:03.321739    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-92.ec2.internal_7615070c530a7fa377bebdacf1747122/haproxy/0.log"
Sep 17 16:46:04.655217 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:04.655176    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-tj7nf/must-gather-8cwfd"]
Sep 17 16:46:04.655698 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:04.655414    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-tj7nf/must-gather-8cwfd" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="copy" containerID="cri-o://2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859" gracePeriod=2
Sep 17 16:46:04.667929 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:04.667758    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-tj7nf/must-gather-8cwfd"]
Sep 17 16:46:04.669218 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:46:04.668971    2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-tj7nf\"/\"must-gather-8cwfd\": pods \"must-gather-8cwfd\" not found" pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:46:05.052886 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.052724    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tj7nf_must-gather-8cwfd_bcda84fb-cb10-4b2a-941b-1a34a1f3b894/copy/0.log"
Sep 17 16:46:05.070320 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.062567    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:46:05.071068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.071032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-must-gather-output\") pod \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") "
Sep 17 16:46:05.071189 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.071120    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8sx9\" (UniqueName: \"kubernetes.io/projected/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-kube-api-access-v8sx9\") pod \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\" (UID: \"bcda84fb-cb10-4b2a-941b-1a34a1f3b894\") "
Sep 17 16:46:05.072661 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.072619    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bcda84fb-cb10-4b2a-941b-1a34a1f3b894" (UID: "bcda84fb-cb10-4b2a-941b-1a34a1f3b894"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:05.094768 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.093238    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-kube-api-access-v8sx9" (OuterVolumeSpecName: "kube-api-access-v8sx9") pod "bcda84fb-cb10-4b2a-941b-1a34a1f3b894" (UID: "bcda84fb-cb10-4b2a-941b-1a34a1f3b894"). InnerVolumeSpecName "kube-api-access-v8sx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:46:05.171782 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.171737    2557 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-must-gather-output\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:46:05.171782 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.171778    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8sx9\" (UniqueName: \"kubernetes.io/projected/bcda84fb-cb10-4b2a-941b-1a34a1f3b894-kube-api-access-v8sx9\") on node \"ip-10-0-140-92.ec2.internal\" DevicePath \"\""
Sep 17 16:46:05.362283 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.362204    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tj7nf_must-gather-8cwfd_bcda84fb-cb10-4b2a-941b-1a34a1f3b894/copy/0.log"
Sep 17 16:46:05.362677 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.362647    2557 generic.go:356] "Generic (PLEG): container finished" podID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerID="2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859" exitCode=143
Sep 17 16:46:05.362798 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.362763    2557 scope.go:117] "RemoveContainer" containerID="2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859"
Sep 17 16:46:05.362955 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.362932    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tj7nf/must-gather-8cwfd"
Sep 17 16:46:05.387177 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.387151    2557 scope.go:117] "RemoveContainer" containerID="c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"
Sep 17 16:46:05.409435 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.409405    2557 scope.go:117] "RemoveContainer" containerID="2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859"
Sep 17 16:46:05.409907 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:46:05.409878    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859\": container with ID starting with 2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859 not found: ID does not exist" containerID="2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859"
Sep 17 16:46:05.410005 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.409922    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859"} err="failed to get container status \"2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859\": rpc error: code = NotFound desc = could not find container \"2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859\": container with ID starting with 2827d719287d3cf06400b8739aeb5cbc1bdce757b883ed6adbf8cb47f58a5859 not found: ID does not exist"
Sep 17 16:46:05.410005 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.409949    2557 scope.go:117] "RemoveContainer" containerID="c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"
Sep 17 16:46:05.410286 ip-10-0-140-92 kubenswrapper[2557]: E0917 16:46:05.410243    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714\": container with ID starting with c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714 not found: ID does not exist" containerID="c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"
Sep 17 16:46:05.410361 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:05.410286    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714"} err="failed to get container status \"c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714\": rpc error: code = NotFound desc = could not find container \"c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714\": container with ID starting with c69f88cd8ca05ea56bdc0ae165282585d543f3b17f19a8bdea019a9262daa714 not found: ID does not exist"
Sep 17 16:46:07.034112 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.034072    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" path="/var/lib/kubelet/pods/bcda84fb-cb10-4b2a-941b-1a34a1f3b894/volumes"
Sep 17 16:46:07.107042 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.106995    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/alertmanager/0.log"
Sep 17 16:46:07.136120 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.136071    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/config-reloader/0.log"
Sep 17 16:46:07.169666 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.169638    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/kube-rbac-proxy-web/0.log"
Sep 17 16:46:07.204571 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.204543    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/kube-rbac-proxy/0.log"
Sep 17 16:46:07.233622 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.233580    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/kube-rbac-proxy-metric/0.log"
Sep 17 16:46:07.262513 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.262485    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/prom-label-proxy/0.log"
Sep 17 16:46:07.302643 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.302532    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b6603c1-8c14-4e55-a970-83de991de4e7/init-config-reloader/0.log"
Sep 17 16:46:07.410502 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.410460    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-g2xtr_8de00618-a4d1-4ec2-9a15-b634ffb3cc89/kube-state-metrics/0.log"
Sep 17 16:46:07.438837 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.438811    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-g2xtr_8de00618-a4d1-4ec2-9a15-b634ffb3cc89/kube-rbac-proxy-main/0.log"
Sep 17 16:46:07.466241 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.466214    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-g2xtr_8de00618-a4d1-4ec2-9a15-b634ffb3cc89/kube-rbac-proxy-self/0.log"
Sep 17 16:46:07.796764 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.796732    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvzxb_e9799fab-3ab4-43fd-a620-4dabcb0f9a6c/node-exporter/0.log"
Sep 17 16:46:07.831662 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.831630    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvzxb_e9799fab-3ab4-43fd-a620-4dabcb0f9a6c/kube-rbac-proxy/0.log"
Sep 17 16:46:07.865804 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.865777    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvzxb_e9799fab-3ab4-43fd-a620-4dabcb0f9a6c/init-textfile/0.log"
Sep 17 16:46:07.901383 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.901343    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2glt4_fc2a21ad-52bd-4169-abde-2d74fbd473ee/kube-rbac-proxy-main/0.log"
Sep 17 16:46:07.929989 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.929953    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2glt4_fc2a21ad-52bd-4169-abde-2d74fbd473ee/kube-rbac-proxy-self/0.log"
Sep 17 16:46:07.955800 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:07.955720    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2glt4_fc2a21ad-52bd-4169-abde-2d74fbd473ee/openshift-state-metrics/0.log"
Sep 17 16:46:08.000523 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.000488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/prometheus/0.log"
Sep 17 16:46:08.035854 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.035829    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/config-reloader/0.log"
Sep 17 16:46:08.074754 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.074726    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/thanos-sidecar/0.log"
Sep 17 16:46:08.106564 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.106531    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/kube-rbac-proxy-web/0.log"
Sep 17 16:46:08.136474 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.136390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/kube-rbac-proxy/0.log"
Sep 17 16:46:08.166541 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.166496    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/kube-rbac-proxy-thanos/0.log"
Sep 17 16:46:08.193667 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.193542    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d943d705-859a-4e5a-9d9d-302d3a318bac/init-config-reloader/0.log"
Sep 17 16:46:08.246050 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.245925    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-zsj2s_0558ae32-24f5-45d5-b55a-4e835c0c9b2d/prometheus-operator/0.log"
Sep 17 16:46:08.280762 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.280731    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-zsj2s_0558ae32-24f5-45d5-b55a-4e835c0c9b2d/kube-rbac-proxy/0.log"
Sep 17 16:46:08.317708 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.317664    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-6l7tx_4794d008-e662-4299-8e12-b4b736b8b915/prometheus-operator-admission-webhook/0.log"
Sep 17 16:46:08.370584 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.370548    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f658797d9-cpzvn_0e64ec39-be7d-4675-aa5b-bd6ff8244541/telemeter-client/0.log"
Sep 17 16:46:08.401486 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.401461    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f658797d9-cpzvn_0e64ec39-be7d-4675-aa5b-bd6ff8244541/reload/0.log"
Sep 17 16:46:08.440107 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:08.440077    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f658797d9-cpzvn_0e64ec39-be7d-4675-aa5b-bd6ff8244541/kube-rbac-proxy/0.log"
Sep 17 16:46:10.875747 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:10.875716    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-xxgdj_d2620d2a-0f4c-4c8d-a6af-6921ec7fbc23/download-server/0.log"
Sep 17 16:46:11.522407 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.522364    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"]
Sep 17 16:46:11.522913 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.522886    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="gather"
Sep 17 16:46:11.522913 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.522914    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="gather"
Sep 17 16:46:11.523110 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.522933    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="copy"
Sep 17 16:46:11.523110 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.522942    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="copy"
Sep 17 16:46:11.523110 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.523020    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="copy"
Sep 17 16:46:11.523110 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.523035    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcda84fb-cb10-4b2a-941b-1a34a1f3b894" containerName="gather"
Sep 17 16:46:11.528497 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.528475    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.537998 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.537973    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"]
Sep 17 16:46:11.633391 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.633345    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-podres\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.633612 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.633444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-sys\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.633612 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.633463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-proc\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.633748 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.633623    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-lib-modules\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.633748 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.633661    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkn5h\" (UniqueName: \"kubernetes.io/projected/17ca812a-f011-4a5e-80e3-75b41fb6ea98-kube-api-access-mkn5h\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735059 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-sys\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735069    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-proc\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735148    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-lib-modules\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkn5h\" (UniqueName: \"kubernetes.io/projected/17ca812a-f011-4a5e-80e3-75b41fb6ea98-kube-api-access-mkn5h\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-sys\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-podres\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735248 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735223    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-proc\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735563 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-lib-modules\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.735563 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.735371    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17ca812a-f011-4a5e-80e3-75b41fb6ea98-podres\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.745827 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.745802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkn5h\" (UniqueName: \"kubernetes.io/projected/17ca812a-f011-4a5e-80e3-75b41fb6ea98-kube-api-access-mkn5h\") pod \"perf-node-gather-daemonset-xtvk8\" (UID: \"17ca812a-f011-4a5e-80e3-75b41fb6ea98\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:11.839085 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:11.839002    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:12.129152 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.129124    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"]
Sep 17 16:46:12.131973 ip-10-0-140-92 kubenswrapper[2557]: W0917 16:46:12.131934    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17ca812a_f011_4a5e_80e3_75b41fb6ea98.slice/crio-d1bca9812b0ac48be3387cb80ee07914813e3ba946d99ef3c859de8f1aa76ce7 WatchSource:0}: Error finding container d1bca9812b0ac48be3387cb80ee07914813e3ba946d99ef3c859de8f1aa76ce7: Status 404 returned error can't find the container with id d1bca9812b0ac48be3387cb80ee07914813e3ba946d99ef3c859de8f1aa76ce7
Sep 17 16:46:12.394402 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.394332    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c6q8r_4024cb32-d700-42bf-9bd5-6217de9aeea6/dns/0.log"
Sep 17 16:46:12.408016 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.407977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8" event={"ID":"17ca812a-f011-4a5e-80e3-75b41fb6ea98","Type":"ContainerStarted","Data":"79a69f39ba29dc4c4bb77cd53ea6922f58e03cd2da45283125639e0f23424f21"}
Sep 17 16:46:12.408016 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.408024    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8" event={"ID":"17ca812a-f011-4a5e-80e3-75b41fb6ea98","Type":"ContainerStarted","Data":"d1bca9812b0ac48be3387cb80ee07914813e3ba946d99ef3c859de8f1aa76ce7"}
Sep 17 16:46:12.408224 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.408114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:12.421356 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.421325    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c6q8r_4024cb32-d700-42bf-9bd5-6217de9aeea6/kube-rbac-proxy/0.log"
Sep 17 16:46:12.426218 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.426154    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8" podStartSLOduration=1.4261329489999999 podStartE2EDuration="1.426132949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:46:12.424838551 +0000 UTC m=+3047.723983050" watchObservedRunningTime="2026-09-17 16:46:12.426132949 +0000 UTC m=+3047.725277427"
Sep 17 16:46:12.501075 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:12.501049    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2dzfn_356ce5ee-7b00-405b-975d-d63f02d9d071/dns-node-resolver/0.log"
Sep 17 16:46:13.098796 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:13.098741    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r74cc_da2f1f6b-54eb-409d-b042-48403b5dbaf2/node-ca/0.log"
Sep 17 16:46:14.294819 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:14.294791    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62qjw_661ed124-a6a6-4713-a554-6e75b33d9221/serve-healthcheck-canary/0.log"
Sep 17 16:46:14.792524 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:14.792496    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42tk6_9097059a-4324-4b3a-880c-ffb932e01953/kube-rbac-proxy/0.log"
Sep 17 16:46:14.848746 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:14.848722    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42tk6_9097059a-4324-4b3a-880c-ffb932e01953/exporter/0.log"
Sep 17 16:46:14.898504 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:14.898472    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42tk6_9097059a-4324-4b3a-880c-ffb932e01953/extractor/0.log"
Sep 17 16:46:17.625767 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:17.625736    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-tcjqv_c3a02fa4-5377-4983-b72b-b47858f70163/s3-init/0.log"
Sep 17 16:46:17.649203 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:17.649180    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-tcjqv_c3a02fa4-5377-4983-b72b-b47858f70163/download-hf-model/0.log"
Sep 17 16:46:17.679167 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:17.679138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-c9sr2_c07d1017-1953-4760-aaa7-f65677043405/seaweedfs/0.log"
Sep 17 16:46:18.421068 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:18.421040    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-xtvk8"
Sep 17 16:46:23.935462 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:23.935432    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:46:23.983460 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:23.983434    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/egress-router-binary-copy/0.log"
Sep 17 16:46:24.029077 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:24.029052    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/cni-plugins/0.log"
Sep 17 16:46:24.074561 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:24.074533    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/bond-cni-plugin/0.log"
Sep 17 16:46:24.121632 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:24.121586    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/routeoverride-cni/0.log"
Sep 17 16:46:24.181730 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:24.181708    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/whereabouts-cni-bincopy/0.log"
Sep 17 16:46:24.235009 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:24.234937    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9zbj_71c49110-1c75-4ad3-8118-23de3f1a05ec/whereabouts-cni/0.log"
Sep 17 16:46:25.277173 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:25.277138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/0.log"
Sep 17 16:46:25.284046 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:25.284027    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r64lr_eb842786-bcb1-4e36-8ccf-2afbff487c28/kube-multus/1.log"
Sep 17 16:46:25.371311 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:25.371284    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6kjfr_e376dd58-f13e-43de-bb4f-e2cac00b537f/network-metrics-daemon/0.log"
Sep 17 16:46:25.403078 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:25.403050    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6kjfr_e376dd58-f13e-43de-bb4f-e2cac00b537f/kube-rbac-proxy/0.log"
Sep 17 16:46:26.719428 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.719404    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-controller/0.log"
Sep 17 16:46:26.762378 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.762350    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/1.log"
Sep 17 16:46:26.762527 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.762428    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovn-acl-logging/0.log"
Sep 17 16:46:26.802745 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.802722    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/kube-rbac-proxy-node/0.log"
Sep 17 16:46:26.850714 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.850685    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:46:26.898980 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.898952    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/northd/0.log"
Sep 17 16:46:26.936888 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.936861    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/nbdb/0.log"
Sep 17 16:46:26.985786 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:26.985709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/sbdb/0.log"
Sep 17 16:46:27.058401 ip-10-0-140-92 kubenswrapper[2557]: I0917 16:46:27.058372    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bnw9m_05678d54-6457-4d4f-996c-72ffa691cb2b/ovnkube-controller/0.log"