Sep 17 15:37:01.496948 ip-10-0-143-66 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:37:01.496962 ip-10-0-143-66 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:37:01.496972 ip-10-0-143-66 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:37:01.497191 ip-10-0-143-66 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:37:11.732761 ip-10-0-143-66 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 15:37:11.732780 ip-10-0-143-66 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7925522f19834978ac9ed941cac5b9c1 --
Sep 17 15:39:06.925512 ip-10-0-143-66 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:39:07.316440 ip-10-0-143-66 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:39:07.316440 ip-10-0-143-66 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:39:07.316440 ip-10-0-143-66 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:39:07.316440 ip-10-0-143-66 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:39:07.316440 ip-10-0-143-66 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:39:07.317405 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.317340    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:39:07.320582 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320567    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:07.320582 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320582    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320588    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320593    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320596    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320599    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320602    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320605    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320608    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320611    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320620    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320623    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320625    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320628    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320631    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320634    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320636    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320639    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320642    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320645    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320647    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:07.320643 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320650    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320653    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320656    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320659    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320662    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320664    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320667    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320669    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320673    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320676    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320679    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320682    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320684    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320687    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320689    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320692    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320695    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320697    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320700    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:07.321115 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320702    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320705    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320708    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320711    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320713    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320716    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320718    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320720    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320723    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320725    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320728    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320730    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320733    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320736    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320738    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320741    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320743    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320746    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320749    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:07.321607 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320751    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320753    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320756    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320758    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320761    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320763    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320766    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320768    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320771    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320773    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320776    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320778    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320782    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320784    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320788    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320791    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320793    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320796    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320798    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320801    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:07.322084 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320804    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320807    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320810    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320812    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320815    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320817    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320820    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320823    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320825    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320828    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320831    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320833    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320836    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320839    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320841    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320845    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320848    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320851    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320854    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.320857    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:07.322583 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321555    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321561    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321564    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321569    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321574    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321577    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321580    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321584    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321587    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321590    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321593    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321595    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321598    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321600    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321603    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321605    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321609    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321612    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321615    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:07.323077 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321618    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321620    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321623    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321626    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321628    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321631    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321633    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321636    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321638    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321641    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321644    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321646    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321649    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321651    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321654    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321657    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321659    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321662    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321664    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321668    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:07.323560 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321671    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321674    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321677    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321680    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321683    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321686    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321688    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321691    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321693    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321696    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321698    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321701    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321703    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321706    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321708    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321711    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321713    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321716    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321718    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321721    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:07.324049 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321723    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321726    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321730    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321734    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321738    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321741    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321744    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321747    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321749    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321752    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321755    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321758    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321761    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321764    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321767    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321770    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321772    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321775    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321778    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321780    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:07.324550 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321783    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321785    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321788    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321791    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321793    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321859    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321864    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321868    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321871    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321873    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321877    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321879    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321882    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321884    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321888    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321891    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321893    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321896    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321899    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:07.325041 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.321901    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322267    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322272    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322276    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322278    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322282    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322285    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322288    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322290    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322293    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322296    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322298    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322301    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322303    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322306    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322309    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322311    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322314    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322316    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322319    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:07.325548 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322322    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322324    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322327    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322330    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322332    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322336    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322340    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322344    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322348    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322351    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322354    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322357    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322360    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322363    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322366    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322369    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322371    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322376    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322379    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:07.326053 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322382    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322385    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322388    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322391    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322393    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322396    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322399    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322401    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322404    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322406    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322409    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322411    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322414    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322416    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322419    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322421    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322424    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322426    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322429    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322431    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:07.326534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322434    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322436    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322439    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322441    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322444    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322447    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322450    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322453    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322455    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322458    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322460    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322463    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322466    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322469    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322471    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322474    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322476    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322479    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322481    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:07.327117 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322488    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322491    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322493    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322496    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322498    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322501    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322503    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322506    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322508    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322511    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322513    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322516    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322519    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322524    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322526    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322529    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322532    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322534    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322537    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322539    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:07.327628 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.322542    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322601    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322632    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322638    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322642    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322647    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322650    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322654    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322658    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322661    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322664    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322667    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322670    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322673    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322677    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322680    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322683    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322685    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322688    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322692    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322694    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322697    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322700    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322704    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:39:07.328133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322709    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322713    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322716    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322719    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322722    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322725    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322728    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322731    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322733    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322737    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322740    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322743    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322746    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322748    2563 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322752    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322756    2563 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322759    2563 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322761    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322764    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322767    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322771    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322774    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322777    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322780    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322783    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:39:07.328726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322785    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322788    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322791    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322794    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322797    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322800    2563 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322803    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322808    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322811    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322815    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322818    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322821    2563 flags.go:64] FLAG: --help="false"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322824    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322827    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322830    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322833    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322837    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322840    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322843    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322846    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322848    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322851    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322856    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322859    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:39:07.329628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322862    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322865    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322868    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322870    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322873    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322876    2563 flags.go:64] FLAG: --lock-file=""
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322879    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322882    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322885    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322890    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322893    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322896    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322899    2563 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322902    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322906    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322910    2563 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322913    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322917    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322920    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322924    2563 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322927    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322930    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322933    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322936    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322939    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:39:07.330690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322943    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322946    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322952    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322955    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322958    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322961    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322965    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322971    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322974    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322977    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322980    2563 flags.go:64] FLAG: --port="10250"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322983    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322986    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8a09bce60dc8bcd"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322989    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322992    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322995    2563 flags.go:64] FLAG: --register-node="true"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.322998    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323002    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323005    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323008    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323010    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323014    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323018    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323021    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323024    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:39:07.331773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323027    2563 flags.go:64] FLAG: --runonce="false"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323030    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323033    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323036    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323039    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323042    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323045    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323048    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323051    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323054    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323057    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323060    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323062    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323065    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323069    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323072    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323077    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323080    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323083    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323086    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323089    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323092    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323095    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323098    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323101    2563 flags.go:64] FLAG: --v="2"
Sep 17 15:39:07.332886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323104    2563 flags.go:64] FLAG: --version="false"
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323108    2563 flags.go:64] FLAG: --vmodule=""
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323111    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.323115    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323224    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323228    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323231    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323234    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323237    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323253    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323256    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323258    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323261    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323265    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323269    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323272    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323274    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323277    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323280    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323282    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:07.333924 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323285    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323287    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323291    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323294    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323296    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323299    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323302    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323304    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323307    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323310    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323312    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323315    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323317    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323320    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323322    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323325    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323331    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323333    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323336    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323339    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:07.334608 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323341    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323344    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323346    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323349    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323352    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323354    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323357    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323359    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323362    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323364    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323367    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323370    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323372    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323375    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323378    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323382    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323386    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323388    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323391    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:07.335102 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323394    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323397    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323401    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323404    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323406    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323409    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323412    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323414    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323417    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323424    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323427    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323429    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323432    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323435    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323438    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323441    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323444    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323446    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323449    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323451    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:07.335575 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323454    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323457    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323460    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323462    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323465    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323467    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323470    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323473    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323475    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323479    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323482    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323484    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323487    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323490    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323493    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323495    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323498    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323500    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323503    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323506    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:07.336052 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323509    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323513    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323516    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.323519    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.324333    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.331638    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.331656    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.331677    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.331685    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331771    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331779    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331784    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331789    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331793    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331798    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:07.336547 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331802    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331809    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331815    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331820    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331825    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331829    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331834    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331838    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331842    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331846    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331850    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331854    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331859    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331863    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331867    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331871    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331875    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331879    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331884    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:07.336914 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331888    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331892    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331896    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331901    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331916    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331921    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331926    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331930    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331934    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331938    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331943    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331947    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331954    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331960    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331965    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331969    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331974    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331979    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331983    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:07.337426 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331988    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331993    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.331998    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332003    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332008    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332012    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332017    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332021    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332026    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332030    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332035    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332039    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332043    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332047    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332051    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332055    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332060    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332065    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332077    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332082    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:07.337913 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332086    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332090    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332094    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332099    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332104    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332108    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332111    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332116    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332120    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332124    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332129    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332134    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332146    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332150    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332154    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332158    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332162    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332167    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332171    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332175    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:07.338408 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332179    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332183    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332188    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332191    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332195    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332200    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332204    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332208    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332212    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332217    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332222    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332233    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332237    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332257    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:07.338896 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332264    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.332272    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332488    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332496    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332501    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332505    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332510    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332514    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332518    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332522    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332526    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332530    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332535    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332539    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332543    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:07.339346 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332547    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332551    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332558    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332565    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332570    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332574    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332579    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332584    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332588    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332593    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332599    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332606    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332610    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332615    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332620    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332632    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332637    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332642    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332646    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:07.339708 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332651    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332656    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332660    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332665    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332669    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332673    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332677    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332681    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332686    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332690    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332694    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332699    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332703    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332707    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332711    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332715    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332719    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332723    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332727    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332732    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:07.340160 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332736    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332740    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332744    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332749    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332753    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332757    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332761    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332764    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332769    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332780    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332785    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332789    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332793    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332797    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332801    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332805    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332809    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332813    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332817    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332822    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:07.340660 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332826    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332830    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332834    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332839    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332845    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332851    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332856    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332861    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332865    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332869    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332873    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332878    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332881    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332885    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332889    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332893    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332897    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332901    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332905    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:07.341145 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332909    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332912    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332917    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332929    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332933    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332937    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332941    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:07.332946    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.332954    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.334762    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.336682    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.337717    2563 server.go:1019] "Starting client certificate rotation"
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.337813    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:39:07.341616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.337863    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:39:07.341952 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.340832    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:39:07.341952 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.341932    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:39:07.353281 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.353264    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:39:07.356559 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.356544    2563 log.go:25] "Validated CRI v1 image API"
Sep 17 15:39:07.357427 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.357408    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:39:07.360199 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.360181    2563 fs.go:135] Filesystem UUIDs: map[07a0c242-a1ae-4be8-b625-c17385565e46:/dev/nvme0n1p4 734f660f-f670-47d9-983f-0e9b40ede263:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 15:39:07.360279 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.360199    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:39:07.366497 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.366396    2563 manager.go:216] Machine: {Timestamp:2026-09-17 15:39:07.365144638 +0000 UTC m=+0.327444194 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099111 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2649b047396d19883d5c48b3406162 SystemUUID:ec2649b0-4739-6d19-883d-5c48b3406162 BootID:7925522f-1983-4978-ac9e-d941cac5b9c1 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:bc:8d:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:bc:8d:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:ec:e1:8c:75:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:39:07.366497 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.366494    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:39:07.366607 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.366559    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:39:07.368917 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.368902    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:39:07.369501 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.369480    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:39:07.369638 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.369504    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:39:07.370724 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.370714    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:39:07.370759 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.370727    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:39:07.370759 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.370735    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:39:07.371322 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.371312    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:39:07.372145 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.372136    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:39:07.375839 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375828    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:39:07.375935 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375927    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:39:07.375972 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375952    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:39:07.375972 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375962    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:39:07.376026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375973    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:39:07.376026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375982    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:39:07.376026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.375990    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:39:07.376296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.376286    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:39:07.376338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.376303    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:39:07.378556 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.378537    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:39:07.380609 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.380595    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:39:07.380661 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.380626    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:39:07.382621 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382581    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:39:07.382621 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382615    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382628    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382641    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382658    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382670    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382697    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382713    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382724    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:39:07.382738 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382734    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:39:07.382946 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382765    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:39:07.382946 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.382774    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:39:07.383423 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.383407    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:39:07.383490 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.383428    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:39:07.387091 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.387075    2563 server.go:1268] "Started kubelet"
Sep 17 15:39:07.387254 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.387185    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:39:07.387333 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.387318    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:39:07.387392 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.387354    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:39:07.387704 ip-10-0-143-66 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:39:07.388652 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.388469    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:39:07.390929 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.390910    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:39:07.391345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.391325    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:39:07.391520 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.391488    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:39:07.391520 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.391496    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:39:07.391636 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.391561    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:39:07.393620 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.393604    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:39:07.393620 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.393617    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:39:07.394163 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.394035    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.394268 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394236    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:39:07.394322 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394269    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:39:07.394415 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394364    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:39:07.394576 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394489    2563 factory.go:55] Registering systemd factory
Sep 17 15:39:07.394576 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394519    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:39:07.394668 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394636    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:39:07.394668 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.394644    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:39:07.395413 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.395397    2563 factory.go:153] Registering CRI-O factory
Sep 17 15:39:07.395413 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.395414    2563 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:39:07.395555 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.395478    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:39:07.395555 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.395549    2563 factory.go:103] Registering Raw factory
Sep 17 15:39:07.395655 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.395565    2563 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:39:07.396316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.396151    2563 manager.go:349] Starting recovery of all containers
Sep 17 15:39:07.401635 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.401613    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:39:07.401726 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.401647    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:39:07.402519 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.402498    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:39:07.402824 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.401724    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e056e449d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.387054546 +0000 UTC m=+0.349354084,LastTimestamp:2026-09-17 15:39:07.387054546 +0000 UTC m=+0.349354084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.409648 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.409628    2563 manager.go:354] Recovery completed
Sep 17 15:39:07.414114 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.414084    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 15:39:07.418690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.418673    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.420736 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.420716    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.420791 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.420752    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.420791 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.420768    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.421376 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.421359    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:39:07.421376 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.421376    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:39:07.421490 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.421393    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:39:07.421490 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.421364    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.422202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.422188    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.422291 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.422216    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.422291 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.422229    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.423564 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.423550    2563 policy_none.go:49] "None policy: Start"
Sep 17 15:39:07.423615 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.423571    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:39:07.423615 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.423584    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:39:07.425179 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.425118    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e058e63e50  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.420737104 +0000 UTC m=+0.383036658,LastTimestamp:2026-09-17 15:39:07.420737104 +0000 UTC m=+0.383036658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.425530 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.425519    2563 policy_none.go:47] "Start"
Sep 17 15:39:07.439716 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.439641    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e058e693ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.420758955 +0000 UTC m=+0.383058503,LastTimestamp:2026-09-17 15:39:07.420758955 +0000 UTC m=+0.383058503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.455549    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e058e6cd21  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.420773665 +0000 UTC m=+0.383073203,LastTimestamp:2026-09-17 15:39:07.420773665 +0000 UTC m=+0.383073203,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.464858    2563 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.464895    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.464906    2563 server.go:87] "Starting device plugin registration server"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.465184    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.465222    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.465298    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.465375    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.465385    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.465825    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:39:07.477266 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.465912    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.477632 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.477271    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-66.ec2.internal.18d625e058e63e50\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e058e63e50  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.420737104 +0000 UTC m=+0.383036658,LastTimestamp:2026-09-17 15:39:07.422200583 +0000 UTC m=+0.384500124,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.486472 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.486408    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-66.ec2.internal.18d625e058e693ab\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e058e693ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.420758955 +0000 UTC m=+0.383058503,LastTimestamp:2026-09-17 15:39:07.422221837 +0000 UTC m=+0.384521379,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.511872 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.511816    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-66.ec2.internal.18d625e058e6cd21\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e058e6cd21  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.420773665 +0000 UTC m=+0.383073203,LastTimestamp:2026-09-17 15:39:07.422234056 +0000 UTC m=+0.384533594,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.527609 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.527546    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-66.ec2.internal.18d625e05ba94fdd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-66.ec2.internal,UID:ip-10-0-143-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-66.ec2.internal,},FirstTimestamp:2026-09-17 15:39:07.467075549 +0000 UTC m=+0.429375073,LastTimestamp:2026-09-17 15:39:07.467075549 +0000 UTC m=+0.429375073,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-66.ec2.internal,}"
Sep 17 15:39:07.528501 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.528483    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:39:07.528566 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.528539    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:39:07.528566 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.528561    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:39:07.528654 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.528612    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:39:07.537986 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.537957    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 15:39:07.559320 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.559301    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8bg5z"
Sep 17 15:39:07.566337 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.566325    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.567546 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.567496    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.567546 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.567519    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.567546 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.567530    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.567546 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.567545    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.571296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.571282    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8bg5z"
Sep 17 15:39:07.588083 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.588068    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.588158 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.588088    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-66.ec2.internal\": node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.616397 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.616379    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.629350 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.629332    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"]
Sep 17 15:39:07.629403 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.629390    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.630650 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.630634    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.630748 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.630659    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.630748 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.630668    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.632721 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.632709    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.632843 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.632830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.632878 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.632856    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.633441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.633422    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.633516 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.633450    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.633516 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.633459    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.633516 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.633422    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.633603 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.633530    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.633603 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.633545    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.635480 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.635465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.635562 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.635487    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:07.636101 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.636088    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:07.636160 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.636108    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:07.636160 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.636117    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:07.653753 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.653732    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-66.ec2.internal\" not found" node="ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.657886 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.657872    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-66.ec2.internal\" not found" node="ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.696723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.696708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52e741d951636f48f8b3d798ae5e6e81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal\" (UID: \"52e741d951636f48f8b3d798ae5e6e81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.696795 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.696731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e741d951636f48f8b3d798ae5e6e81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal\" (UID: \"52e741d951636f48f8b3d798ae5e6e81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.696795 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.696747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fbe400037057f056594337561e74444-config\") pod \"kube-apiserver-proxy-ip-10-0-143-66.ec2.internal\" (UID: \"9fbe400037057f056594337561e74444\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.716810 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.716794    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.797649 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.797624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52e741d951636f48f8b3d798ae5e6e81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal\" (UID: \"52e741d951636f48f8b3d798ae5e6e81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.797729 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.797695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e741d951636f48f8b3d798ae5e6e81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal\" (UID: \"52e741d951636f48f8b3d798ae5e6e81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.797729 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.797656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e741d951636f48f8b3d798ae5e6e81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal\" (UID: \"52e741d951636f48f8b3d798ae5e6e81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.797812 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.797732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52e741d951636f48f8b3d798ae5e6e81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal\" (UID: \"52e741d951636f48f8b3d798ae5e6e81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.797812 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.797736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fbe400037057f056594337561e74444-config\") pod \"kube-apiserver-proxy-ip-10-0-143-66.ec2.internal\" (UID: \"9fbe400037057f056594337561e74444\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.797812 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.797775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fbe400037057f056594337561e74444-config\") pod \"kube-apiserver-proxy-ip-10-0-143-66.ec2.internal\" (UID: \"9fbe400037057f056594337561e74444\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.817779 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.817733    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.918497 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:07.918472    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:07.956636 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.956619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:07.960145 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:07.960125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:08.019053 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.019027    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.119652 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.119621    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.220850 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.220816    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.243634 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.243609    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:08.321345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.321313    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.337681 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.337652    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:39:08.337807 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.337788    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:39:08.387519 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.387491    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:08.421544 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.421518    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.426729 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.426709    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:08.428098 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.428066    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:08.428172 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.428117    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:08.428172 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.428134    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:08.522361 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.522336    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.576404 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.576368    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:34:07 +0000 UTC" deadline="2028-05-14 12:49:26.089669319 +0000 UTC"
Sep 17 15:39:08.576404 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.576403    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14517h10m17.513271056s"
Sep 17 15:39:08.580422 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:08.580386    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbe400037057f056594337561e74444.slice/crio-810689a4b5bb09acf3a1c641f998a8e50687597cf145f45fca86056916cbcad9 WatchSource:0}: Error finding container 810689a4b5bb09acf3a1c641f998a8e50687597cf145f45fca86056916cbcad9: Status 404 returned error can't find the container with id 810689a4b5bb09acf3a1c641f998a8e50687597cf145f45fca86056916cbcad9
Sep 17 15:39:08.580674 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:08.580652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e741d951636f48f8b3d798ae5e6e81.slice/crio-177d195cd9ee9763674db0de7efe3509b085dd952cda9ce17bd6808e543d2bd8 WatchSource:0}: Error finding container 177d195cd9ee9763674db0de7efe3509b085dd952cda9ce17bd6808e543d2bd8: Status 404 returned error can't find the container with id 177d195cd9ee9763674db0de7efe3509b085dd952cda9ce17bd6808e543d2bd8
Sep 17 15:39:08.585719 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.585696    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:39:08.622961 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.622933    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.724081 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.724026    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.824989 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:08.824962    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-66.ec2.internal\" not found"
Sep 17 15:39:08.915418 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.915393    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:08.928627 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.928603    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:08.994166 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:08.994105    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:09.009363 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.009203    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:39:09.010853 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.010622    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal"
Sep 17 15:39:09.027503 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.027484    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:39:09.377467 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.377431    2563 apiserver.go:52] "Watching apiserver"
Sep 17 15:39:09.385821 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.385794    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:39:09.386216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.386191    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-78bdg","kube-system/konnectivity-agent-bpphv","kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rxlnn","openshift-dns/node-resolver-wmrjn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal","openshift-multus/multus-2l64w","openshift-multus/network-metrics-daemon-stkm5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl","openshift-image-registry/node-ca-z8vfk","openshift-multus/multus-additional-cni-plugins-24wtg","openshift-network-diagnostics/network-check-target-xpcqf","openshift-network-operator/iptables-alerter-hjvd2","openshift-ovn-kubernetes/ovnkube-node-j5hdl"]
Sep 17 15:39:09.389108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.389084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.391326 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.391305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.391954 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.391921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.392110 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.392039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:39:09.392295 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.392269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:39:09.392383 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.392324    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.392548 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.392482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l2tjr\""
Sep 17 15:39:09.393729 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.393688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.394891 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.394870    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.395034 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.395017    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:39:09.395252 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.395233    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.396035 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.396014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:39:09.396154 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.396137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pfrlt\""
Sep 17 15:39:09.396441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.396408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.396536 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.396486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:39:09.396536 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.396500    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s7ctq\""
Sep 17 15:39:09.399440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.399394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-znc77\""
Sep 17 15:39:09.399440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.399422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.399595 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.399529    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.400434 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.400362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.403364 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.402862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.403364 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.402992    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:39:09.403364 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.403333    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.403559 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.403389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-28vfc\""
Sep 17 15:39:09.403559 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.403418    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.405294 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.405273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.405385 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.405365    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.406959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-cni-multus\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.406992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzvr\" (UniqueName: \"kubernetes.io/projected/b75028fb-ee74-4933-9375-41d26937c9f7-kube-api-access-xzzvr\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-run\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-modprobe-d\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysconfig\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b75028fb-ee74-4933-9375-41d26937c9f7-cni-binary-copy\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-cni-bin\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-kubelet\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d0b166d-7a54-4772-beea-06c71d2a67fe-hosts-file\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80276513-b7ca-446e-b5c4-db0572ae2be6-agent-certs\") pod \"konnectivity-agent-bpphv\" (UID: \"80276513-b7ca-446e-b5c4-db0572ae2be6\") " pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-system-cni-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-hostroot\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b75028fb-ee74-4933-9375-41d26937c9f7-multus-daemon-config\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-etc-kubernetes\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-var-lib-kubelet\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.407500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bng56\" (UniqueName: \"kubernetes.io/projected/a401461f-8dbc-457f-9eb1-844e01558af0-kube-api-access-bng56\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-cni-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-multus-certs\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0b166d-7a54-4772-beea-06c71d2a67fe-tmp-dir\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysctl-conf\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a401461f-8dbc-457f-9eb1-844e01558af0-tmp\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysctl-d\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-lib-modules\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-netns\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-conf-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80276513-b7ca-446e-b5c4-db0572ae2be6-konnectivity-ca\") pod \"konnectivity-agent-bpphv\" (UID: \"80276513-b7ca-446e-b5c4-db0572ae2be6\") " pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-systemd\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-sys\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a401461f-8dbc-457f-9eb1-844e01558af0-etc-tuned\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-socket-dir-parent\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-k8s-cni-cncf-io\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.408601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwlz\" (UniqueName: \"kubernetes.io/projected/2d0b166d-7a54-4772-beea-06c71d2a67fe-kube-api-access-slwlz\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.409509 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.407995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-kubernetes\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.409509 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.408019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-host\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.409509 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.408043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-cnibin\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.409509 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.408146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pxtfd\""
Sep 17 15:39:09.410857 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.410349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.410857 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.410708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.410857 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.410815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.411065 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.411036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jvbjw\""
Sep 17 15:39:09.411354 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.411296    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:39:09.411532 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.411513    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:39:09.413131 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.413086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:09.415221 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.413461    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:09.415221 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.414633    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m42g7\""
Sep 17 15:39:09.415221 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.414674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:39:09.415491 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.415284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:39:09.417703 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.417681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:09.417800 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.417770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:09.420795 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.420767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.424113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.424097    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:39:09.424277 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.424161    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2n42d\""
Sep 17 15:39:09.424359 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.424163    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.424359 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.424225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:39:09.424741 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.424720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:39:09.424827 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.424746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.425252 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.425215    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:39:09.445012 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.444991    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4znkx"
Sep 17 15:39:09.454063 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.454044    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4znkx"
Sep 17 15:39:09.495583 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.495563    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:39:09.508986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.508958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzvr\" (UniqueName: \"kubernetes.io/projected/b75028fb-ee74-4933-9375-41d26937c9f7-kube-api-access-xzzvr\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509093 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.508989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-cni-bin\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509093 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d0b166d-7a54-4772-beea-06c71d2a67fe-hosts-file\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.509093 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134bc838-5236-4aee-adc0-05b1b7d57fde-host-slash\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.509093 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-device-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d0b166d-7a54-4772-beea-06c71d2a67fe-hosts-file\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-cni-bin\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2aee588-9d72-44b7-99be-919ddc5cec56-ovn-node-metrics-cert\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-system-cni-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bng56\" (UniqueName: \"kubernetes.io/projected/a401461f-8dbc-457f-9eb1-844e01558af0-kube-api-access-bng56\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-slash\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-system-cni-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-ovnkube-config\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-cni-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0b166d-7a54-4772-beea-06c71d2a67fe-tmp-dir\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysctl-conf\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cnibin\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-cni-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-lib-modules\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134bc838-5236-4aee-adc0-05b1b7d57fde-iptables-alerter-script\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-netns\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-conf-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysctl-conf\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80276513-b7ca-446e-b5c4-db0572ae2be6-konnectivity-ca\") pod \"konnectivity-agent-bpphv\" (UID: \"80276513-b7ca-446e-b5c4-db0572ae2be6\") " pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-netns\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-sys\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-conf-dir\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.509790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a401461f-8dbc-457f-9eb1-844e01558af0-etc-tuned\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-lib-modules\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpm5q\" (UniqueName: \"kubernetes.io/projected/134bc838-5236-4aee-adc0-05b1b7d57fde-kube-api-access-vpm5q\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-sys\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-node-log\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0b166d-7a54-4772-beea-06c71d2a67fe-tmp-dir\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-k8s-cni-cncf-io\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwlz\" (UniqueName: \"kubernetes.io/projected/2d0b166d-7a54-4772-beea-06c71d2a67fe-kube-api-access-slwlz\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-kubernetes\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-k8s-cni-cncf-io\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.509987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-host\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-run-netns\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-ovnkube-script-lib\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cni-binary-copy\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.510639 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-cni-multus\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-run\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-serviceca\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-kubernetes\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4x5\" (UniqueName: \"kubernetes.io/projected/932cded3-c571-411f-b440-19ac1bd83639-kube-api-access-mw4x5\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-host\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-systemd\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-ovn\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-cni-netd\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-modprobe-d\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-cni-multus\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysconfig\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b75028fb-ee74-4933-9375-41d26937c9f7-cni-binary-copy\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-run\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-kubelet\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.511389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80276513-b7ca-446e-b5c4-db0572ae2be6-agent-certs\") pod \"konnectivity-agent-bpphv\" (UID: \"80276513-b7ca-446e-b5c4-db0572ae2be6\") " pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzn6t\" (UniqueName: \"kubernetes.io/projected/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-kube-api-access-hzn6t\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-var-lib-kubelet\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-registration-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-cni-bin\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80276513-b7ca-446e-b5c4-db0572ae2be6-konnectivity-ca\") pod \"konnectivity-agent-bpphv\" (UID: \"80276513-b7ca-446e-b5c4-db0572ae2be6\") " pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510640    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-modprobe-d\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysconfig\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-hostroot\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-hostroot\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b75028fb-ee74-4933-9375-41d26937c9f7-multus-daemon-config\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-etc-kubernetes\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.510953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-var-lib-kubelet\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-system-cni-dir\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-etc-kubernetes\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-var-lib-kubelet\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkj2\" (UniqueName: \"kubernetes.io/projected/b2aee588-9d72-44b7-99be-919ddc5cec56-kube-api-access-7xkj2\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.512231 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-multus-certs\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b75028fb-ee74-4933-9375-41d26937c9f7-cni-binary-copy\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a401461f-8dbc-457f-9eb1-844e01558af0-tmp\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-host\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-host-run-multus-certs\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f68z6\" (UniqueName: \"kubernetes.io/projected/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-kube-api-access-f68z6\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-etc-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-log-socket\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysctl-d\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b75028fb-ee74-4933-9375-41d26937c9f7-multus-daemon-config\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6369f6c9-96e2-4131-a71f-a778cee270d2-kubelet-config\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-sys-fs\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqtd\" (UniqueName: \"kubernetes.io/projected/49b4d7d5-be21-40d0-a148-e82943fb1e96-kube-api-access-5wqtd\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-sysctl-d\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-systemd-units\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-var-lib-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.513124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-env-overrides\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-systemd\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a401461f-8dbc-457f-9eb1-844e01558af0-etc-systemd\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-socket-dir-parent\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6369f6c9-96e2-4131-a71f-a778cee270d2-dbus\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-multus-socket-dir-parent\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-socket-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-cnibin\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.511934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-kubelet\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.514030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.512035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b75028fb-ee74-4933-9375-41d26937c9f7-cnibin\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.515376 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.515179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a401461f-8dbc-457f-9eb1-844e01558af0-etc-tuned\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.515559 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.515536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80276513-b7ca-446e-b5c4-db0572ae2be6-agent-certs\") pod \"konnectivity-agent-bpphv\" (UID: \"80276513-b7ca-446e-b5c4-db0572ae2be6\") " pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.515633 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.515584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a401461f-8dbc-457f-9eb1-844e01558af0-tmp\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.521865 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.521843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bng56\" (UniqueName: \"kubernetes.io/projected/a401461f-8dbc-457f-9eb1-844e01558af0-kube-api-access-bng56\") pod \"tuned-rxlnn\" (UID: \"a401461f-8dbc-457f-9eb1-844e01558af0\") " pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.521951 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.521923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzvr\" (UniqueName: \"kubernetes.io/projected/b75028fb-ee74-4933-9375-41d26937c9f7-kube-api-access-xzzvr\") pod \"multus-2l64w\" (UID: \"b75028fb-ee74-4933-9375-41d26937c9f7\") " pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.522590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.522564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwlz\" (UniqueName: \"kubernetes.io/projected/2d0b166d-7a54-4772-beea-06c71d2a67fe-kube-api-access-slwlz\") pod \"node-resolver-wmrjn\" (UID: \"2d0b166d-7a54-4772-beea-06c71d2a67fe\") " pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.533705 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.533656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal" event={"ID":"52e741d951636f48f8b3d798ae5e6e81","Type":"ContainerStarted","Data":"177d195cd9ee9763674db0de7efe3509b085dd952cda9ce17bd6808e543d2bd8"}
Sep 17 15:39:09.534931 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.534901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal" event={"ID":"9fbe400037057f056594337561e74444","Type":"ContainerStarted","Data":"810689a4b5bb09acf3a1c641f998a8e50687597cf145f45fca86056916cbcad9"}
Sep 17 15:39:09.613171 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpm5q\" (UniqueName: \"kubernetes.io/projected/134bc838-5236-4aee-adc0-05b1b7d57fde-kube-api-access-vpm5q\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.613309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.613309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-node-log\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.613496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-run-netns\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-ovnkube-script-lib\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.613496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cni-binary-copy\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.613496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.613664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-node-log\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-serviceca\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.613664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:09.613664 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.613630    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:09.613664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4x5\" (UniqueName: \"kubernetes.io/projected/932cded3-c571-411f-b440-19ac1bd83639-kube-api-access-mw4x5\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:09.613870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-systemd\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613870 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.613716    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:10.113674392 +0000 UTC m=+3.075973946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:09.613870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-ovn\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-systemd\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-cni-netd\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.613870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzn6t\" (UniqueName: \"kubernetes.io/projected/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-kube-api-access-hzn6t\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-run-netns\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-ovnkube-script-lib\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.613974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-ovn\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-registration-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-cni-bin\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-cni-netd\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-system-cni-dir\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkj2\" (UniqueName: \"kubernetes.io/projected/b2aee588-9d72-44b7-99be-919ddc5cec56-kube-api-access-7xkj2\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-cni-bin\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-host\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-system-cni-dir\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-registration-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f68z6\" (UniqueName: \"kubernetes.io/projected/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-kube-api-access-f68z6\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-host\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-etc-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.614288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-log-socket\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6369f6c9-96e2-4131-a71f-a778cee270d2-kubelet-config\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-sys-fs\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqtd\" (UniqueName: \"kubernetes.io/projected/49b4d7d5-be21-40d0-a148-e82943fb1e96-kube-api-access-5wqtd\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-systemd-units\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-log-socket\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-var-lib-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-etc-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.614399    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-sys-fs\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.614459    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:10.114439738 +0000 UTC m=+3.076739285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-env-overrides\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6369f6c9-96e2-4131-a71f-a778cee270d2-kubelet-config\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-systemd-units\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6369f6c9-96e2-4131-a71f-a778cee270d2-dbus\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-var-lib-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-socket-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-kubelet\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134bc838-5236-4aee-adc0-05b1b7d57fde-host-slash\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6369f6c9-96e2-4131-a71f-a778cee270d2-dbus\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-device-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-kubelet\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/134bc838-5236-4aee-adc0-05b1b7d57fde-host-slash\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-device-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49b4d7d5-be21-40d0-a148-e82943fb1e96-socket-dir\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-serviceca\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.615883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-run-openvswitch\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2aee588-9d72-44b7-99be-919ddc5cec56-ovn-node-metrics-cert\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-env-overrides\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-slash\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-ovnkube-config\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cnibin\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134bc838-5236-4aee-adc0-05b1b7d57fde-iptables-alerter-script\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2aee588-9d72-44b7-99be-919ddc5cec56-host-slash\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.614993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cnibin\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.615058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-cni-binary-copy\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.615475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2aee588-9d72-44b7-99be-919ddc5cec56-ovnkube-config\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.615503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.616500 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.615555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/134bc838-5236-4aee-adc0-05b1b7d57fde-iptables-alerter-script\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.617316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.617288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2aee588-9d72-44b7-99be-919ddc5cec56-ovn-node-metrics-cert\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.626191 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.626158    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:09.626291 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.626194    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:09.626291 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.626209    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:09.626291 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:09.626275    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:10.12625556 +0000 UTC m=+3.088555106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:09.627645 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.627575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4x5\" (UniqueName: \"kubernetes.io/projected/932cded3-c571-411f-b440-19ac1bd83639-kube-api-access-mw4x5\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:09.627740 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.627690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f68z6\" (UniqueName: \"kubernetes.io/projected/e57adf70-aba9-4a36-a5ee-7ffdf90a6110-kube-api-access-f68z6\") pod \"multus-additional-cni-plugins-24wtg\" (UID: \"e57adf70-aba9-4a36-a5ee-7ffdf90a6110\") " pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.627980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.627954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpm5q\" (UniqueName: \"kubernetes.io/projected/134bc838-5236-4aee-adc0-05b1b7d57fde-kube-api-access-vpm5q\") pod \"iptables-alerter-hjvd2\" (UID: \"134bc838-5236-4aee-adc0-05b1b7d57fde\") " pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.629251 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.629228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzn6t\" (UniqueName: \"kubernetes.io/projected/b9a93a6a-8f86-4e35-83f1-a55ff30da13f-kube-api-access-hzn6t\") pod \"node-ca-z8vfk\" (UID: \"b9a93a6a-8f86-4e35-83f1-a55ff30da13f\") " pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.629411 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.629336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqtd\" (UniqueName: \"kubernetes.io/projected/49b4d7d5-be21-40d0-a148-e82943fb1e96-kube-api-access-5wqtd\") pod \"aws-ebs-csi-driver-node-bwvhl\" (UID: \"49b4d7d5-be21-40d0-a148-e82943fb1e96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.630525 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.630507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkj2\" (UniqueName: \"kubernetes.io/projected/b2aee588-9d72-44b7-99be-919ddc5cec56-kube-api-access-7xkj2\") pod \"ovnkube-node-j5hdl\" (UID: \"b2aee588-9d72-44b7-99be-919ddc5cec56\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:09.702617 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.702576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l64w"
Sep 17 15:39:09.713460 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.713439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmrjn"
Sep 17 15:39:09.726351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.726329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:09.733140 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.733117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rxlnn"
Sep 17 15:39:09.741532 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.741516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hjvd2"
Sep 17 15:39:09.748840 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.748820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl"
Sep 17 15:39:09.755793 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.755777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z8vfk"
Sep 17 15:39:09.761380 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.761362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-24wtg"
Sep 17 15:39:09.767204 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:09.767184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:10.118689 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.118621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:10.118689 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.118684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:10.118856 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.118767    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:10.118856 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.118770    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:10.118856 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.118819    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:11.118804161 +0000 UTC m=+4.081103692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:10.118856 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.118834    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:11.118827313 +0000 UTC m=+4.081126838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:10.219489 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.219457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:10.219648 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.219636    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:10.219701 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.219655    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:10.219701 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.219665    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:10.219796 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.219736    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:11.219719257 +0000 UTC m=+4.182018797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:10.457768 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.457731    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:34:09 +0000 UTC" deadline="2028-05-15 13:35:17.00962048 +0000 UTC"
Sep 17 15:39:10.457768 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.457759    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14541h56m6.551865732s"
Sep 17 15:39:10.484196 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.484082    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0b166d_7a54_4772_beea_06c71d2a67fe.slice/crio-15ff8ffe502478a877c38ed8620a112e6ebb24e00d517edcbf8d0e258c5fb026 WatchSource:0}: Error finding container 15ff8ffe502478a877c38ed8620a112e6ebb24e00d517edcbf8d0e258c5fb026: Status 404 returned error can't find the container with id 15ff8ffe502478a877c38ed8620a112e6ebb24e00d517edcbf8d0e258c5fb026
Sep 17 15:39:10.486599 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.486547    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75028fb_ee74_4933_9375_41d26937c9f7.slice/crio-b61be5cd89f032691ec33851f950ce4d808c88a09aa7d716de396b15a3d2ca13 WatchSource:0}: Error finding container b61be5cd89f032691ec33851f950ce4d808c88a09aa7d716de396b15a3d2ca13: Status 404 returned error can't find the container with id b61be5cd89f032691ec33851f950ce4d808c88a09aa7d716de396b15a3d2ca13
Sep 17 15:39:10.488968 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.488948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b4d7d5_be21_40d0_a148_e82943fb1e96.slice/crio-20895a2dbf4063f0b81a045e30987900107f0442470bb69bd36fe1445a8105ef WatchSource:0}: Error finding container 20895a2dbf4063f0b81a045e30987900107f0442470bb69bd36fe1445a8105ef: Status 404 returned error can't find the container with id 20895a2dbf4063f0b81a045e30987900107f0442470bb69bd36fe1445a8105ef
Sep 17 15:39:10.512465 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.512436    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda401461f_8dbc_457f_9eb1_844e01558af0.slice/crio-42fe2c0d863b794d07c4ab022db769f352f622bf0fa07bdc66dcba6ff9c13b45 WatchSource:0}: Error finding container 42fe2c0d863b794d07c4ab022db769f352f622bf0fa07bdc66dcba6ff9c13b45: Status 404 returned error can't find the container with id 42fe2c0d863b794d07c4ab022db769f352f622bf0fa07bdc66dcba6ff9c13b45
Sep 17 15:39:10.513465 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.513445    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134bc838_5236_4aee_adc0_05b1b7d57fde.slice/crio-115e723f32049878adb4479def88b9fe7bc7700a01e50c9487889b75e5ac62ae WatchSource:0}: Error finding container 115e723f32049878adb4479def88b9fe7bc7700a01e50c9487889b75e5ac62ae: Status 404 returned error can't find the container with id 115e723f32049878adb4479def88b9fe7bc7700a01e50c9487889b75e5ac62ae
Sep 17 15:39:10.514891 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.514862    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57adf70_aba9_4a36_a5ee_7ffdf90a6110.slice/crio-620d82fb17feb32047a7e3f02b0f893e69f9b8dfe7c2c5813482914ba27e7732 WatchSource:0}: Error finding container 620d82fb17feb32047a7e3f02b0f893e69f9b8dfe7c2c5813482914ba27e7732: Status 404 returned error can't find the container with id 620d82fb17feb32047a7e3f02b0f893e69f9b8dfe7c2c5813482914ba27e7732
Sep 17 15:39:10.515769 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.515553    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2aee588_9d72_44b7_99be_919ddc5cec56.slice/crio-de7e700395010323a5533d0c156511b6742e7e4de6cc2dd9627c74e529d9986a WatchSource:0}: Error finding container de7e700395010323a5533d0c156511b6742e7e4de6cc2dd9627c74e529d9986a: Status 404 returned error can't find the container with id de7e700395010323a5533d0c156511b6742e7e4de6cc2dd9627c74e529d9986a
Sep 17 15:39:10.517557 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.517398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a93a6a_8f86_4e35_83f1_a55ff30da13f.slice/crio-720b769480fd85eb16a630408b407060fa213f92036f2374b6f7cd200fdf0d04 WatchSource:0}: Error finding container 720b769480fd85eb16a630408b407060fa213f92036f2374b6f7cd200fdf0d04: Status 404 returned error can't find the container with id 720b769480fd85eb16a630408b407060fa213f92036f2374b6f7cd200fdf0d04
Sep 17 15:39:10.517938 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:10.517902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80276513_b7ca_446e_b5c4_db0572ae2be6.slice/crio-83a4621be353249a6a4aacf4773c8405f1e96d28062f96c1f5bd78fbeb0800ab WatchSource:0}: Error finding container 83a4621be353249a6a4aacf4773c8405f1e96d28062f96c1f5bd78fbeb0800ab: Status 404 returned error can't find the container with id 83a4621be353249a6a4aacf4773c8405f1e96d28062f96c1f5bd78fbeb0800ab
Sep 17 15:39:10.529579 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.529559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:10.529694 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:10.529671    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:10.538790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.538764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmrjn" event={"ID":"2d0b166d-7a54-4772-beea-06c71d2a67fe","Type":"ContainerStarted","Data":"15ff8ffe502478a877c38ed8620a112e6ebb24e00d517edcbf8d0e258c5fb026"}
Sep 17 15:39:10.539985 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.539956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bpphv" event={"ID":"80276513-b7ca-446e-b5c4-db0572ae2be6","Type":"ContainerStarted","Data":"83a4621be353249a6a4aacf4773c8405f1e96d28062f96c1f5bd78fbeb0800ab"}
Sep 17 15:39:10.541223 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.541182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z8vfk" event={"ID":"b9a93a6a-8f86-4e35-83f1-a55ff30da13f","Type":"ContainerStarted","Data":"720b769480fd85eb16a630408b407060fa213f92036f2374b6f7cd200fdf0d04"}
Sep 17 15:39:10.542598 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.542570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"de7e700395010323a5533d0c156511b6742e7e4de6cc2dd9627c74e529d9986a"}
Sep 17 15:39:10.543998 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.543975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rxlnn" event={"ID":"a401461f-8dbc-457f-9eb1-844e01558af0","Type":"ContainerStarted","Data":"42fe2c0d863b794d07c4ab022db769f352f622bf0fa07bdc66dcba6ff9c13b45"}
Sep 17 15:39:10.545624 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.545592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl" event={"ID":"49b4d7d5-be21-40d0-a148-e82943fb1e96","Type":"ContainerStarted","Data":"20895a2dbf4063f0b81a045e30987900107f0442470bb69bd36fe1445a8105ef"}
Sep 17 15:39:10.546878 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.546851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l64w" event={"ID":"b75028fb-ee74-4933-9375-41d26937c9f7","Type":"ContainerStarted","Data":"b61be5cd89f032691ec33851f950ce4d808c88a09aa7d716de396b15a3d2ca13"}
Sep 17 15:39:10.550188 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.550164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerStarted","Data":"620d82fb17feb32047a7e3f02b0f893e69f9b8dfe7c2c5813482914ba27e7732"}
Sep 17 15:39:10.557218 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:10.557195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hjvd2" event={"ID":"134bc838-5236-4aee-adc0-05b1b7d57fde","Type":"ContainerStarted","Data":"115e723f32049878adb4479def88b9fe7bc7700a01e50c9487889b75e5ac62ae"}
Sep 17 15:39:11.127705 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.126906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:11.127705 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.127000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:11.127705 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.127129    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:11.127705 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.127193    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:13.127174561 +0000 UTC m=+6.089474102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:11.127705 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.127613    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:11.127705 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.127670    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:13.127652858 +0000 UTC m=+6.089952383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:11.227819 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.227776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:11.227990 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.227975    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:11.228068 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.227998    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:11.228068 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.228011    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:11.228068 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.228062    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:13.228045702 +0000 UTC m=+6.190345234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:11.349593 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.349565    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:11.458538 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.458459    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:34:09 +0000 UTC" deadline="2028-05-26 05:32:59.329428142 +0000 UTC"
Sep 17 15:39:11.458538 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.458495    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14797h53m47.870937625s"
Sep 17 15:39:11.529632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.529469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:11.529632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.529473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:11.529632 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.529586    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:11.529883 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:11.529670    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:11.587298 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.586680    2563 generic.go:356] "Generic (PLEG): container finished" podID="52e741d951636f48f8b3d798ae5e6e81" containerID="aafc634f12eef606099783395c7f3ff28e46920b9639b6ae06a839bd294de194" exitCode=0
Sep 17 15:39:11.587298 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.586951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal" event={"ID":"52e741d951636f48f8b3d798ae5e6e81","Type":"ContainerDied","Data":"aafc634f12eef606099783395c7f3ff28e46920b9639b6ae06a839bd294de194"}
Sep 17 15:39:11.609134 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.608476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal" event={"ID":"9fbe400037057f056594337561e74444","Type":"ContainerStarted","Data":"6b8f4b129e957bd505daa3438366ea1d7e414e74d1d798e4f476ed67d84e3f37"}
Sep 17 15:39:11.627237 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:11.627183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-66.ec2.internal" podStartSLOduration=2.627165957 podStartE2EDuration="2.627165957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:11.626597388 +0000 UTC m=+4.588896958" watchObservedRunningTime="2026-09-17 15:39:11.627165957 +0000 UTC m=+4.589465504"
Sep 17 15:39:12.529706 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:12.529668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:12.530305 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:12.529799    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:12.618512 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:12.618477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal" event={"ID":"52e741d951636f48f8b3d798ae5e6e81","Type":"ContainerStarted","Data":"59398bf49ae76454d31166a0de695a4fb3256b8652333558d4017c250359b504"}
Sep 17 15:39:13.142847 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:13.142811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:13.143042 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:13.142862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:13.143042 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.143002    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:13.143159 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.143059    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:17.143040737 +0000 UTC m=+10.105340266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:13.143159 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.143126    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:13.143308 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.143162    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:17.14314893 +0000 UTC m=+10.105448460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:13.243588 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:13.243462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:13.243753 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.243666    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:13.243753 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.243688    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:13.243753 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.243701    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:13.243753 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.243750    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:17.243734711 +0000 UTC m=+10.206034249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:13.530525 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:13.530393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:13.530525 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:13.530466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:13.530993 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.530593    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:13.530993 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:13.530691    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:14.529175 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:14.529142    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:14.529420 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:14.529277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:15.529161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:15.529129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:15.529161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:15.529151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:15.529623 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:15.529280    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:15.529623 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:15.529432    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:16.528956 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:16.528924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:16.529136 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:16.529051    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:17.176757 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:17.176706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:17.176757 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:17.176756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:17.177317 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.176893    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:17.177317 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.176954    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:25.176934663 +0000 UTC m=+18.139234201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:17.177436 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.177372    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:17.177436 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.177421    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:25.177406039 +0000 UTC m=+18.139705577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:17.277530 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:17.277493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:17.277677 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.277651    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:17.277742 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.277677    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:17.277742 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.277690    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:17.277842 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.277750    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:25.27773189 +0000 UTC m=+18.240031436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:17.531149 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:17.530432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:17.531149 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.530755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:17.531149 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:17.530863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:17.531149 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:17.530975    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:18.529172 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:18.529141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:18.529589 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:18.529277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:19.528987 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:19.528955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:19.529158 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:19.529078    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:19.529158 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:19.529123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:19.529561 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:19.529214    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:20.529663 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:20.529629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:20.530042 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:20.529738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:21.529140 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:21.529108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:21.529140 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:21.529140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:21.529407 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:21.529259    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:21.529407 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:21.529348    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:22.529089 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:22.529058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:22.529538 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:22.529171    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:23.529228 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:23.529197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:23.529626 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:23.529324    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:23.529626 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:23.529399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:23.529626 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:23.529499    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:24.529722 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:24.529690    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:24.530279 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:24.529810    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:25.240209 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:25.240173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:25.240209 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:25.240220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:25.240453 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.240341    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:25.240453 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.240391    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:41.240376881 +0000 UTC m=+34.202676418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:25.240453 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.240340    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:25.240453 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.240456    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:41.240444 +0000 UTC m=+34.202743537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:25.340806 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:25.340776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:25.340982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.340946    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:25.340982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.340969    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:25.340982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.340983    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:25.341131 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.341040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:41.341021686 +0000 UTC m=+34.303321225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:25.529063 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:25.528995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:25.529230 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.529118    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:25.529230 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:25.529174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:25.529335 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:25.529291    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:26.529317 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:26.529288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:26.529709 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:26.529392    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:27.530332 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:27.530297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:27.530739 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:27.530414    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:27.530739 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:27.530498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:27.530739 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:27.530599    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:28.529319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.529276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:28.529415 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:28.529394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:28.649528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.649500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl" event={"ID":"49b4d7d5-be21-40d0-a148-e82943fb1e96","Type":"ContainerStarted","Data":"87ec6bccff90b4b1d3d626a428b0ff804e7f25258b6a6260d45bae555e93bd19"}
Sep 17 15:39:28.651053 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.651028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l64w" event={"ID":"b75028fb-ee74-4933-9375-41d26937c9f7","Type":"ContainerStarted","Data":"eec530408784869291fd292c32eb9f85805f475c4154171757daaa9cabfc800c"}
Sep 17 15:39:28.652752 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.652715    2563 generic.go:356] "Generic (PLEG): container finished" podID="e57adf70-aba9-4a36-a5ee-7ffdf90a6110" containerID="fc9a41cb22eb4597931c9d7a2ff3a186457b6de2525ec68a5e57230a7d665b8c" exitCode=0
Sep 17 15:39:28.652851 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.652768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerDied","Data":"fc9a41cb22eb4597931c9d7a2ff3a186457b6de2525ec68a5e57230a7d665b8c"}
Sep 17 15:39:28.657169 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.656128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmrjn" event={"ID":"2d0b166d-7a54-4772-beea-06c71d2a67fe","Type":"ContainerStarted","Data":"5e1e209a34a6325079d52a67c2e590ffd0c96d52d9481f600cfba24fced45553"}
Sep 17 15:39:28.660743 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.660276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bpphv" event={"ID":"80276513-b7ca-446e-b5c4-db0572ae2be6","Type":"ContainerStarted","Data":"d5234f4963b8cc41a0fbe7b2e54c03471d50a5207e50bccffc949b5b1ec78ab0"}
Sep 17 15:39:28.664313 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.664286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z8vfk" event={"ID":"b9a93a6a-8f86-4e35-83f1-a55ff30da13f","Type":"ContainerStarted","Data":"e87ae7efdf68bb5211bdec6df326858db4eb99bb309216ed54081023613c232e"}
Sep 17 15:39:28.667545 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.667520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"3c188ff84dc12ab0ca13d0a4594c723bceb68819a6f04eb37b600dcd60bf08f7"}
Sep 17 15:39:28.667652 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.667597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"f56ec94154b6c3c4a75f0f28d6478ac9d1959cad195911b5615ca306e36bcb4f"}
Sep 17 15:39:28.667652 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.667649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"1f7dc012a4311c4f9f5fc7643a53bc60de3332614873ccab2c2c385b1ccd40af"}
Sep 17 15:39:28.667759 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.667697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"c52cff413eb9ed9ad7b99385deb4cbac6e3f2c4d47851258d3abf0b21c512727"}
Sep 17 15:39:28.667759 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.667744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"f6405dbec28b44d942a08b5f17c4e7c6a08311408d4fdac4919eaa84a1037f8f"}
Sep 17 15:39:28.669455 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.669423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rxlnn" event={"ID":"a401461f-8dbc-457f-9eb1-844e01558af0","Type":"ContainerStarted","Data":"7d3f62b11b12e938b35519131d421c8f3f017daf35a04c48ae95aca5564bd196"}
Sep 17 15:39:28.676864 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.676822    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-66.ec2.internal" podStartSLOduration=19.676806484 podStartE2EDuration="19.676806484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:12.649491741 +0000 UTC m=+5.611791300" watchObservedRunningTime="2026-09-17 15:39:28.676806484 +0000 UTC m=+21.639106032"
Sep 17 15:39:28.695680 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.695636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wmrjn" podStartSLOduration=4.147713791 podStartE2EDuration="21.695619859s" totalImagesPullingTime="17.547906068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:28.695413348 +0000 UTC m=+21.657712923" watchObservedRunningTime="2026-09-17 15:39:28.695619859 +0000 UTC m=+21.657919406"
Sep 17 15:39:28.711294 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.711234    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bpphv" podStartSLOduration=8.245262241 podStartE2EDuration="21.711218498s" totalImagesPullingTime="13.465956257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:28.711068514 +0000 UTC m=+21.673368075" watchObservedRunningTime="2026-09-17 15:39:28.711218498 +0000 UTC m=+21.673518046"
Sep 17 15:39:28.732977 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.732941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rxlnn" podStartSLOduration=4.209706639 podStartE2EDuration="21.732929982s" totalImagesPullingTime="17.523223343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:28.731734375 +0000 UTC m=+21.694033952" watchObservedRunningTime="2026-09-17 15:39:28.732929982 +0000 UTC m=+21.695229528"
Sep 17 15:39:28.747800 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.747762    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z8vfk" podStartSLOduration=3.235757767 podStartE2EDuration="20.747751441s" totalImagesPullingTime="17.511993674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:28.747339932 +0000 UTC m=+21.709639503" watchObservedRunningTime="2026-09-17 15:39:28.747751441 +0000 UTC m=+21.710050990"
Sep 17 15:39:28.767081 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:28.767030    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2l64w" podStartSLOduration=4.202463836 podStartE2EDuration="21.767016015s" totalImagesPullingTime="17.564552179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:28.766743022 +0000 UTC m=+21.729042571" watchObservedRunningTime="2026-09-17 15:39:28.767016015 +0000 UTC m=+21.729315562"
Sep 17 15:39:29.515598 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.515429    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:39:29.529504 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.529475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:29.529608 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.529475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:29.529608 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:29.529591    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:29.529756 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:29.529634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:29.675987 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.675895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"d06ff78a72c03f86e176382f07ae23d95f66a025ca66c5dc117962fd2685cdb6"}
Sep 17 15:39:29.678060 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.678030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl" event={"ID":"49b4d7d5-be21-40d0-a148-e82943fb1e96","Type":"ContainerStarted","Data":"f1c189dec3dd8080e09a139bb332dea1042726786b00ddb6a373b9655ca7e5a0"}
Sep 17 15:39:29.679778 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.679754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hjvd2" event={"ID":"134bc838-5236-4aee-adc0-05b1b7d57fde","Type":"ContainerStarted","Data":"c1fa9aa5cddf3cb1a58ad751ae520a2bf1268de47bdeec70fb41735bfc36dbdb"}
Sep 17 15:39:29.698915 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:29.698869    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hjvd2" podStartSLOduration=4.18079437 podStartE2EDuration="21.698851618s" totalImagesPullingTime="17.518057248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:29.698682949 +0000 UTC m=+22.660982506" watchObservedRunningTime="2026-09-17 15:39:29.698851618 +0000 UTC m=+22.661151166"
Sep 17 15:39:30.475920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:30.475809    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:39:29.515595831Z","UUID":"b4d42b6b-ca98-4432-909b-8f31e6c44c1f","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:39:30.477534 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:30.477511    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:39:30.477641 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:30.477549    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:39:30.528828 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:30.528798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:30.528971 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:30.528924    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:30.683994 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:30.683962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl" event={"ID":"49b4d7d5-be21-40d0-a148-e82943fb1e96","Type":"ContainerStarted","Data":"7da314416cd9ffd26126ed7620829aaafe508b3b000ec5280d7c5abb40b92dac"}
Sep 17 15:39:31.529859 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:31.529827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:31.529859 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:31.529844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:31.530168 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:31.529941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:31.530168 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:31.530077    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:31.690611 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:31.690579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"63019d50c8b13dd405f31f0b4ee06ef6d151d23ca3032b75dd6e1d82276af039"}
Sep 17 15:39:32.529638 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:32.529605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:32.529813 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:32.529715    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:32.939966 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:32.939942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:32.940481 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:32.940466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:32.964511 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:32.964475    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwvhl" podStartSLOduration=5.346050171 podStartE2EDuration="24.964461199s" totalImagesPullingTime="19.618411028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:30.708393984 +0000 UTC m=+23.670693556" watchObservedRunningTime="2026-09-17 15:39:32.964461199 +0000 UTC m=+25.926760751"
Sep 17 15:39:33.529143 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.528905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:33.529315 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.529006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:33.529315 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:33.529182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:33.529388 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:33.529313    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:33.696555 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.696525    2563 generic.go:356] "Generic (PLEG): container finished" podID="e57adf70-aba9-4a36-a5ee-7ffdf90a6110" containerID="d6bf16e9de0a9efbcc4280f8ae7e49ae232cf8fd0e0f3895e2614987f63ef8c8" exitCode=0
Sep 17 15:39:33.696720 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.696620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerDied","Data":"d6bf16e9de0a9efbcc4280f8ae7e49ae232cf8fd0e0f3895e2614987f63ef8c8"}
Sep 17 15:39:33.700991 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.700967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" event={"ID":"b2aee588-9d72-44b7-99be-919ddc5cec56","Type":"ContainerStarted","Data":"fc05ec52bf4b7473dac691f739a97f5bf2f77941c3b793ab32a5eaf6ef07b961"}
Sep 17 15:39:33.701256 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.701227    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:33.701708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.701688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bpphv"
Sep 17 15:39:33.747959 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:33.747919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl" podStartSLOduration=8.029862816 podStartE2EDuration="25.747907262s" totalImagesPullingTime="17.718044446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:33.747494769 +0000 UTC m=+26.709794325" watchObservedRunningTime="2026-09-17 15:39:33.747907262 +0000 UTC m=+26.710206807"
Sep 17 15:39:34.529746 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.529724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:34.530287 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:34.529821    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:34.705622 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.705390    2563 generic.go:356] "Generic (PLEG): container finished" podID="e57adf70-aba9-4a36-a5ee-7ffdf90a6110" containerID="e80e6539b2239c25a16f12f43551ca8e187bb7c344fd0dd9f12033e2dee59b07" exitCode=0
Sep 17 15:39:34.705771 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.705449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerDied","Data":"e80e6539b2239c25a16f12f43551ca8e187bb7c344fd0dd9f12033e2dee59b07"}
Sep 17 15:39:34.706373 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.706213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:34.706373 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.706351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:34.706535 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.706405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:34.723625 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.723565    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:34.723743 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.723701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:39:34.985850 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.985769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-78bdg"]
Sep 17 15:39:34.985956 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:34.985903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:34.985998 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:34.985979    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:35.001264 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:35.001227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpcqf"]
Sep 17 15:39:35.001368 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:35.001328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:35.001432 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:35.001413    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:35.023026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:35.022997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stkm5"]
Sep 17 15:39:35.023116 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:35.023092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:35.023200 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:35.023183    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:35.710566 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:35.710537    2563 generic.go:356] "Generic (PLEG): container finished" podID="e57adf70-aba9-4a36-a5ee-7ffdf90a6110" containerID="1ce25c1cf444d2b252f56cd94e1629118456734da661a4a73bf46a91cb5c23ae" exitCode=0
Sep 17 15:39:35.710924 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:35.710627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerDied","Data":"1ce25c1cf444d2b252f56cd94e1629118456734da661a4a73bf46a91cb5c23ae"}
Sep 17 15:39:36.529398 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:36.529372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:36.529550 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:36.529470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:36.529550 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:36.529484    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:36.529653 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:36.529565    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:36.529653 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:36.529572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:36.529653 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:36.529640    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:38.528787 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:38.528750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:38.529292 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:38.528880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:38.529292 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:38.528887    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:38.529292 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:38.528923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:38.529292 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:38.529027    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:38.529292 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:38.529102    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:40.529296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.529228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:40.529591 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.529299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:40.529591 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.529392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:40.529591 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:40.529399    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpcqf" podUID="d7fbee30-abee-40da-9b15-9602c5e4195c"
Sep 17 15:39:40.529591 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:40.529494    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stkm5" podUID="932cded3-c571-411f-b440-19ac1bd83639"
Sep 17 15:39:40.529591 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:40.529572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78bdg" podUID="6369f6c9-96e2-4131-a71f-a778cee270d2"
Sep 17 15:39:40.724967 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.724942    2563 generic.go:356] "Generic (PLEG): container finished" podID="e57adf70-aba9-4a36-a5ee-7ffdf90a6110" containerID="1868754f529ac5f4884b7116c2e0a5edd970258d4748b10e80c6aaad9d51aeb0" exitCode=0
Sep 17 15:39:40.725086 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.724986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerDied","Data":"1868754f529ac5f4884b7116c2e0a5edd970258d4748b10e80c6aaad9d51aeb0"}
Sep 17 15:39:40.747844 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.747828    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-66.ec2.internal" event="NodeReady"
Sep 17 15:39:40.747949 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.747933    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:39:40.807967 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.807945    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dcff994bc-brfvh"]
Sep 17 15:39:40.828937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.828917    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v97nd"]
Sep 17 15:39:40.829068 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.829050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.831616 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.831595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:39:40.831780 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.831726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:39:40.831780 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.831715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:39:40.831892 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.831868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wq7zh\""
Sep 17 15:39:40.839047 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.838695    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:39:40.841170 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.841150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4zz5s"]
Sep 17 15:39:40.841299 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.841258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:40.843585 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.843566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:39:40.843989 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.843970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p86vs\""
Sep 17 15:39:40.844253 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.844225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:39:40.844353 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.844260    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:39:40.855655 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.855636    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcff994bc-brfvh"]
Sep 17 15:39:40.855737 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.855663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v97nd"]
Sep 17 15:39:40.855737 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.855677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4zz5s"]
Sep 17 15:39:40.855806 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.855760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:40.858001 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.857975    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:39:40.858001 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.857996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxcdv\""
Sep 17 15:39:40.858157 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.857977    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:39:40.954540 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-installation-pull-secrets\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.954661 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-image-registry-private-configuration\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.954661 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-trusted-ca\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.954661 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-tmp-dir\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:40.954661 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44603772-8a08-4435-a283-dda01bf4bafa-cert\") pod \"ingress-canary-v97nd\" (UID: \"44603772-8a08-4435-a283-dda01bf4bafa\") " pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:40.954845 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-certificates\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.954845 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-bound-sa-token\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.954845 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:40.954845 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddc8\" (UniqueName: \"kubernetes.io/projected/44603772-8a08-4435-a283-dda01bf4bafa-kube-api-access-dddc8\") pod \"ingress-canary-v97nd\" (UID: \"44603772-8a08-4435-a283-dda01bf4bafa\") " pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:40.955062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.955062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djj8z\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-kube-api-access-djj8z\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.955062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmvn\" (UniqueName: \"kubernetes.io/projected/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-kube-api-access-tqmvn\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:40.955062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa0a7c8e-f752-455c-95c0-94426646eb76-ca-trust-extracted\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:40.955062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:40.954999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-config-volume\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.055787 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-image-registry-private-configuration\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.055787 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-trusted-ca\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.055787 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-tmp-dir\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.056020 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44603772-8a08-4435-a283-dda01bf4bafa-cert\") pod \"ingress-canary-v97nd\" (UID: \"44603772-8a08-4435-a283-dda01bf4bafa\") " pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:41.056020 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-certificates\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056020 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-bound-sa-token\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056020 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.055860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddc8\" (UniqueName: \"kubernetes.io/projected/44603772-8a08-4435-a283-dda01bf4bafa-kube-api-access-dddc8\") pod \"ingress-canary-v97nd\" (UID: \"44603772-8a08-4435-a283-dda01bf4bafa\") " pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.056139    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djj8z\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-kube-api-access-djj8z\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-tmp-dir\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmvn\" (UniqueName: \"kubernetes.io/projected/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-kube-api-access-tqmvn\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.056216 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.056205    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:41.556182005 +0000 UTC m=+34.518481547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.056308    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.056328    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa0a7c8e-f752-455c-95c0-94426646eb76-ca-trust-extracted\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.056388    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:41.556373776 +0000 UTC m=+34.518673300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-config-volume\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-installation-pull-secrets\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-certificates\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.056993 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.056695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa0a7c8e-f752-455c-95c0-94426646eb76-ca-trust-extracted\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.057432 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.057033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-config-volume\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.057555 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.057533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-trusted-ca\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.059968 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.059948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-installation-pull-secrets\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.059968 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.059959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-image-registry-private-configuration\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.060083 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.060051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44603772-8a08-4435-a283-dda01bf4bafa-cert\") pod \"ingress-canary-v97nd\" (UID: \"44603772-8a08-4435-a283-dda01bf4bafa\") " pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:41.068059 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.068031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djj8z\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-kube-api-access-djj8z\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.068643 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.068624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddc8\" (UniqueName: \"kubernetes.io/projected/44603772-8a08-4435-a283-dda01bf4bafa-kube-api-access-dddc8\") pod \"ingress-canary-v97nd\" (UID: \"44603772-8a08-4435-a283-dda01bf4bafa\") " pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:41.068750 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.068646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmvn\" (UniqueName: \"kubernetes.io/projected/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-kube-api-access-tqmvn\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.068750 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.068667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-bound-sa-token\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.150400 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.150379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v97nd"
Sep 17 15:39:41.258731 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.258698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:41.258860 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.258739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:41.258860 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.258843    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:41.258929 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.258868    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:41.258929 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.258909    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret podName:6369f6c9-96e2-4131-a71f-a778cee270d2 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.258891561 +0000 UTC m=+66.221191084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret") pod "global-pull-secret-syncer-78bdg" (UID: "6369f6c9-96e2-4131-a71f-a778cee270d2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:41.258929 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.258922    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.258916474 +0000 UTC m=+66.221215997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:41.359491 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.359462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:41.359627 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.359611    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:41.359667 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.359631    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:41.359667 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.359640    2563 projected.go:196] Error preparing data for projected volume kube-api-access-d7kjc for pod openshift-network-diagnostics/network-check-target-xpcqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:41.359731 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.359688    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc podName:d7fbee30-abee-40da-9b15-9602c5e4195c nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.359672214 +0000 UTC m=+66.321971738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7kjc" (UniqueName: "kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc") pod "network-check-target-xpcqf" (UID: "d7fbee30-abee-40da-9b15-9602c5e4195c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:41.465220 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.465195    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v97nd"]
Sep 17 15:39:41.469161 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:39:41.469137    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44603772_8a08_4435_a283_dda01bf4bafa.slice/crio-882b2e809b57d10ad7329df545269849c3d41cea7d0aa3db3255e989c8b08825 WatchSource:0}: Error finding container 882b2e809b57d10ad7329df545269849c3d41cea7d0aa3db3255e989c8b08825: Status 404 returned error can't find the container with id 882b2e809b57d10ad7329df545269849c3d41cea7d0aa3db3255e989c8b08825
Sep 17 15:39:41.561441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.561422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:41.561982 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.561457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:41.561982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.561555    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:41.561982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.561598    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:42.56158687 +0000 UTC m=+35.523886399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:41.561982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.561556    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:41.561982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.561648    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:39:41.561982 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:41.561697    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:42.561680772 +0000 UTC m=+35.523980312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:39:41.733143 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.733116    2563 generic.go:356] "Generic (PLEG): container finished" podID="e57adf70-aba9-4a36-a5ee-7ffdf90a6110" containerID="b975191f7dcd95392078477ddbd3cc349c021f0aa0c2cfd1064ab8d10d0810e2" exitCode=0
Sep 17 15:39:41.733381 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.733185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerDied","Data":"b975191f7dcd95392078477ddbd3cc349c021f0aa0c2cfd1064ab8d10d0810e2"}
Sep 17 15:39:41.734444 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:41.734419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v97nd" event={"ID":"44603772-8a08-4435-a283-dda01bf4bafa","Type":"ContainerStarted","Data":"882b2e809b57d10ad7329df545269849c3d41cea7d0aa3db3255e989c8b08825"}
Sep 17 15:39:42.529601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.529359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:39:42.529776 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.529409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:39:42.529776 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.529415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:39:42.532664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.532643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:42.533632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.533612    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:42.533751 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.533615    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nm7x2\""
Sep 17 15:39:42.533751 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.533738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:42.533928 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.533910    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:42.534097 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.534018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm2sj\""
Sep 17 15:39:42.569994 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.569971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:42.570318 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.570065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:42.570318 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:42.570123    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:42.570318 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:42.570136    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:39:42.570318 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:42.570164    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:42.570318 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:42.570193    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:44.570178062 +0000 UTC m=+37.532477602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:39:42.570318 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:42.570217    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:44.570203198 +0000 UTC m=+37.532502745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:42.741164 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.741134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24wtg" event={"ID":"e57adf70-aba9-4a36-a5ee-7ffdf90a6110","Type":"ContainerStarted","Data":"65afe42d7e318b549bed8d661d9067a144587f4b3030fd82b99a2c57270e0c0d"}
Sep 17 15:39:42.771976 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:42.771937    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-24wtg" podStartSLOduration=6.822231984 podStartE2EDuration="34.771922448s" totalImagesPullingTime="27.949690464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:42.770390498 +0000 UTC m=+35.732690056" watchObservedRunningTime="2026-09-17 15:39:42.771922448 +0000 UTC m=+35.734221992"
Sep 17 15:39:44.585300 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:44.585270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:44.585669 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:44.585318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:44.585669 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:44.585411    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:44.585669 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:44.585424    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:44.585669 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:44.585439    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:39:44.585669 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:44.585469    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:48.585455398 +0000 UTC m=+41.547754927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:44.585669 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:44.585482    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:48.585476259 +0000 UTC m=+41.547775783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:39:44.746196 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:44.746167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v97nd" event={"ID":"44603772-8a08-4435-a283-dda01bf4bafa","Type":"ContainerStarted","Data":"bbdf77d63ae1da4d6b94bd148fb94cdf1723c77b4bd55c7cdc80d85944fb50bd"}
Sep 17 15:39:44.768662 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:44.768588    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v97nd" podStartSLOduration=2.578010918 podStartE2EDuration="4.768574451s" totalImagesPullingTime="2.190563533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:44.767068531 +0000 UTC m=+37.729368087" watchObservedRunningTime="2026-09-17 15:39:44.768574451 +0000 UTC m=+37.730874031"
Sep 17 15:39:48.609932 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:48.609894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:48.610345 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:48.609945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:48.610345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:48.610052    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:48.610345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:48.610072    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:48.610345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:48.610083    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:39:48.610345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:48.610123    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:56.61010275 +0000 UTC m=+49.572402288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:48.610345 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:48.610136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:56.610130224 +0000 UTC m=+49.572429747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:39:56.664418 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:56.664370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:39:56.664843 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:39:56.664434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:39:56.664843 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:56.664523    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:56.664843 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:56.664595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:12.66457555 +0000 UTC m=+65.626875074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:56.664843 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:56.664600    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:56.664843 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:56.664621    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:39:56.664843 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:39:56.664691    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:12.664674796 +0000 UTC m=+65.626974325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:40:06.723066 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:06.723031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5hdl"
Sep 17 15:40:12.673955 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:12.673904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:40:12.673955 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:12.673960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:40:12.674525 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:12.674072    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:40:12.674525 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:12.674093    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:40:12.674525 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:12.674105    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcff994bc-brfvh: secret "image-registry-tls" not found
Sep 17 15:40:12.674525 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:12.674141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls podName:0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:44.674125248 +0000 UTC m=+97.636424830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls") pod "dns-default-4zz5s" (UID: "0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:12.674525 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:12.674163    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls podName:fa0a7c8e-f752-455c-95c0-94426646eb76 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:44.674150498 +0000 UTC m=+97.636450022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls") pod "image-registry-5dcff994bc-brfvh" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76") : secret "image-registry-tls" not found
Sep 17 15:40:13.278287 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.278230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:40:13.278287 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.278290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:40:13.281133 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.281109    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:40:13.281329 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.281286    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:40:13.289269 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:13.289233    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:40:13.289387 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:13.289320    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs podName:932cded3-c571-411f-b440-19ac1bd83639 nodeName:}" failed. No retries permitted until 2026-09-17 15:41:17.289298591 +0000 UTC m=+130.251598129 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs") pod "network-metrics-daemon-stkm5" (UID: "932cded3-c571-411f-b440-19ac1bd83639") : secret "metrics-daemon-secret" not found
Sep 17 15:40:13.292897 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.292874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6369f6c9-96e2-4131-a71f-a778cee270d2-original-pull-secret\") pod \"global-pull-secret-syncer-78bdg\" (UID: \"6369f6c9-96e2-4131-a71f-a778cee270d2\") " pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:40:13.379009 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.378982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:40:13.382036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.382019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:40:13.391962 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.391945    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:40:13.402398 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.402374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kjc\" (UniqueName: \"kubernetes.io/projected/d7fbee30-abee-40da-9b15-9602c5e4195c-kube-api-access-d7kjc\") pod \"network-check-target-xpcqf\" (UID: \"d7fbee30-abee-40da-9b15-9602c5e4195c\") " pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:40:13.448401 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.448378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78bdg"
Sep 17 15:40:13.456238 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.456217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nm7x2\""
Sep 17 15:40:13.464465 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.464447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:40:13.740754 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.740729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-78bdg"]
Sep 17 15:40:13.743903 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:13.743872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6369f6c9_96e2_4131_a71f_a778cee270d2.slice/crio-85ee53946c3c9d823316b63b69b843d67881f3e88b0dd7ba942cdd0c5d29533d WatchSource:0}: Error finding container 85ee53946c3c9d823316b63b69b843d67881f3e88b0dd7ba942cdd0c5d29533d: Status 404 returned error can't find the container with id 85ee53946c3c9d823316b63b69b843d67881f3e88b0dd7ba942cdd0c5d29533d
Sep 17 15:40:13.758464 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.758435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpcqf"]
Sep 17 15:40:13.761883 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:13.761854    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fbee30_abee_40da_9b15_9602c5e4195c.slice/crio-a90de1cba03db9056bd7ee256a188b77532bb541f14b559e3ae8f51e3666ca1f WatchSource:0}: Error finding container a90de1cba03db9056bd7ee256a188b77532bb541f14b559e3ae8f51e3666ca1f: Status 404 returned error can't find the container with id a90de1cba03db9056bd7ee256a188b77532bb541f14b559e3ae8f51e3666ca1f
Sep 17 15:40:13.816229 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.816198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpcqf" event={"ID":"d7fbee30-abee-40da-9b15-9602c5e4195c","Type":"ContainerStarted","Data":"a90de1cba03db9056bd7ee256a188b77532bb541f14b559e3ae8f51e3666ca1f"}
Sep 17 15:40:13.817345 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:13.817325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-78bdg" event={"ID":"6369f6c9-96e2-4131-a71f-a778cee270d2","Type":"ContainerStarted","Data":"85ee53946c3c9d823316b63b69b843d67881f3e88b0dd7ba942cdd0c5d29533d"}
Sep 17 15:40:18.833617 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:18.833571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpcqf" event={"ID":"d7fbee30-abee-40da-9b15-9602c5e4195c","Type":"ContainerStarted","Data":"b4b1a13832d992d073ae9507f44f192e152fe86eebdf44c8186df559c6581e27"}
Sep 17 15:40:18.834099 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:18.833690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:40:18.835502 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:18.835471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-78bdg" event={"ID":"6369f6c9-96e2-4131-a71f-a778cee270d2","Type":"ContainerStarted","Data":"98eb68aa38117d301aa86703b69d6cbdbff0b989dd98c2dd577321d016343fcc"}
Sep 17 15:40:18.851483 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:18.851424    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xpcqf" podStartSLOduration=66.570941968 podStartE2EDuration="1m10.851409237s" totalImagesPullingTime="4.280467269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:18.850641381 +0000 UTC m=+71.812940943" watchObservedRunningTime="2026-09-17 15:40:18.851409237 +0000 UTC m=+71.813708783"
Sep 17 15:40:18.866440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:18.866398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-78bdg" podStartSLOduration=66.563989432 podStartE2EDuration="1m10.866385608s" totalImagesPullingTime="4.302396176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:18.866317158 +0000 UTC m=+71.828616714" watchObservedRunningTime="2026-09-17 15:40:18.866385608 +0000 UTC m=+71.828685155"
Sep 17 15:40:19.623827 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.623796    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wvvp2"]
Sep 17 15:40:19.627053 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.627027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.631048 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.631028    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:40:19.631152 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.631047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:40:19.631852 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.631835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:40:19.641003 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.640978    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pdt22\""
Sep 17 15:40:19.643712 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.643691    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:40:19.659264 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.659209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvvp2"]
Sep 17 15:40:19.724089 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.724066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-crio-socket\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.724182 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.724115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4kc\" (UniqueName: \"kubernetes.io/projected/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-api-access-tw4kc\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.724182 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.724168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.724282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.724213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-data-volume\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.724325 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.724289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825526 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-crio-socket\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kc\" (UniqueName: \"kubernetes.io/projected/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-api-access-tw4kc\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-data-volume\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-crio-socket\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825745 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.825745 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:19.825705    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:19.825745 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:19.825729    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:19.825851 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:19.825762    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.325747205 +0000 UTC m=+73.288046730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:19.825904 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:19.825854    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.325831182 +0000 UTC m=+73.288130710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:19.825995 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.825978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-data-volume\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:19.836680 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:19.836657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4kc\" (UniqueName: \"kubernetes.io/projected/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-api-access-tw4kc\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:20.329312 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:20.329285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:20.329428 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:20.329335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:20.329428 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:20.329420    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.329505 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:20.329430    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.329505 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:20.329481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.329467526 +0000 UTC m=+74.291767050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.329505 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:20.329500    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.329492374 +0000 UTC m=+74.291791898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:21.005983 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.005950    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"]
Sep 17 15:40:21.008808 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.008788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"
Sep 17 15:40:21.012041 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.012018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-85ks2\""
Sep 17 15:40:21.013028 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.013005    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:21.013137 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.013015    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:21.022180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.022155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"]
Sep 17 15:40:21.034522 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.034497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h64f\" (UniqueName: \"kubernetes.io/projected/69a4cf25-5cba-4821-a187-6042870523ad-kube-api-access-9h64f\") pod \"volume-data-source-validator-86695b7cc7-hssqr\" (UID: \"69a4cf25-5cba-4821-a187-6042870523ad\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"
Sep 17 15:40:21.133740 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.133709    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-578cf489b4-4hnf6"]
Sep 17 15:40:21.134965 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.134933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h64f\" (UniqueName: \"kubernetes.io/projected/69a4cf25-5cba-4821-a187-6042870523ad-kube-api-access-9h64f\") pod \"volume-data-source-validator-86695b7cc7-hssqr\" (UID: \"69a4cf25-5cba-4821-a187-6042870523ad\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"
Sep 17 15:40:21.136676 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.136657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.139890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 15:40:21.139890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 15:40:21.139890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 15:40:21.139890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139774    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 15:40:21.139890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t7ngd\""
Sep 17 15:40:21.140194 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139942    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 15:40:21.140194 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.139993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 15:40:21.145913 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.145886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h64f\" (UniqueName: \"kubernetes.io/projected/69a4cf25-5cba-4821-a187-6042870523ad-kube-api-access-9h64f\") pod \"volume-data-source-validator-86695b7cc7-hssqr\" (UID: \"69a4cf25-5cba-4821-a187-6042870523ad\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"
Sep 17 15:40:21.150574 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.150548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-578cf489b4-4hnf6"]
Sep 17 15:40:21.235904 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.235875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-default-certificate\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.235904 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.235904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.236065 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.235973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-stats-auth\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.236065 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.236001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wrz\" (UniqueName: \"kubernetes.io/projected/a6726151-1a0b-4092-a4f3-5356c7d72b22-kube-api-access-47wrz\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.236065 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.236021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.318105 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.318048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"
Sep 17 15:40:21.336492 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-stats-auth\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.336587 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wrz\" (UniqueName: \"kubernetes.io/projected/a6726151-1a0b-4092-a4f3-5356c7d72b22-kube-api-access-47wrz\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.336587 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.336587 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:21.336587 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-default-certificate\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.336790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.336790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.336658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:21.336790 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336693    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:21.336790 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336758    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.836739038 +0000 UTC m=+74.799038582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:21.336790 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336783    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.33677333 +0000 UTC m=+76.299072864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:21.336790 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336760    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:21.337019 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.336810342 +0000 UTC m=+76.299109882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:21.337019 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336818    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:21.337019 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.336869    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.836848717 +0000 UTC m=+74.799148246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : secret "router-metrics-certs-default" not found
Sep 17 15:40:21.339090 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.339069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-stats-auth\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.339203 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.339172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-default-certificate\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.350826 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.350799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wrz\" (UniqueName: \"kubernetes.io/projected/a6726151-1a0b-4092-a4f3-5356c7d72b22-kube-api-access-47wrz\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.591438 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.591370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr"]
Sep 17 15:40:21.593792 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:21.593771    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a4cf25_5cba_4821_a187_6042870523ad.slice/crio-17acac3d735b5998acfd439b137a9a31c9b6c10e52ea63febf592c8d9bf5ca3e WatchSource:0}: Error finding container 17acac3d735b5998acfd439b137a9a31c9b6c10e52ea63febf592c8d9bf5ca3e: Status 404 returned error can't find the container with id 17acac3d735b5998acfd439b137a9a31c9b6c10e52ea63febf592c8d9bf5ca3e
Sep 17 15:40:21.753788 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.753762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmrjn_2d0b166d-7a54-4772-beea-06c71d2a67fe/dns-node-resolver/0.log"
Sep 17 15:40:21.839331 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.839300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.839448 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.839414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:21.839507 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.839461    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:21.839555 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.839525    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:22.839509358 +0000 UTC m=+75.801808882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : secret "router-metrics-certs-default" not found
Sep 17 15:40:21.839609 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:21.839575    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:22.839556433 +0000 UTC m=+75.801856027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:21.843508 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:21.843442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr" event={"ID":"69a4cf25-5cba-4821-a187-6042870523ad","Type":"ContainerStarted","Data":"17acac3d735b5998acfd439b137a9a31c9b6c10e52ea63febf592c8d9bf5ca3e"}
Sep 17 15:40:22.554161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:22.554134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z8vfk_b9a93a6a-8f86-4e35-83f1-a55ff30da13f/node-ca/0.log"
Sep 17 15:40:22.845921 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:22.845832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:22.845921 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:22.845906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:22.846107 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:22.846033    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:24.846009414 +0000 UTC m=+77.808308939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:22.846107 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:22.846039    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:22.846107 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:22.846087    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:24.846075415 +0000 UTC m=+77.808374941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : secret "router-metrics-certs-default" not found
Sep 17 15:40:22.847569 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:22.847540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr" event={"ID":"69a4cf25-5cba-4821-a187-6042870523ad","Type":"ContainerStarted","Data":"558ce152adeb591c683167ddcfccd2483242d75f3c2c0b76e3f46a2e7e615328"}
Sep 17 15:40:22.865793 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:22.865732    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hssqr" podStartSLOduration=2.034801192 podStartE2EDuration="2.865717257s" totalImagesPullingTime="830.916065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:22.865167409 +0000 UTC m=+75.827466965" watchObservedRunningTime="2026-09-17 15:40:22.865717257 +0000 UTC m=+75.828016859"
Sep 17 15:40:22.954014 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:22.953989    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v97nd_44603772-8a08-4435-a283-dda01bf4bafa/serve-healthcheck-canary/0.log"
Sep 17 15:40:23.349560 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:23.349520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:23.349678 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:23.349593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:23.349678 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:23.349657    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:23.349747 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:23.349679    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:23.349747 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:23.349717    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:27.349700761 +0000 UTC m=+80.312000284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:23.349747 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:23.349733    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:27.349726187 +0000 UTC m=+80.312025711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:24.860405 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:24.860373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:24.860755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:24.860435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:24.860755 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:24.860511    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:24.860755 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:24.860524    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.860510176 +0000 UTC m=+81.822809700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:24.860755 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:24.860558    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.86054677 +0000 UTC m=+81.822846294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : secret "router-metrics-certs-default" not found
Sep 17 15:40:27.377883 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:27.377848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:27.378237 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:27.377937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:27.378237 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:27.377969    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:27.378237 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:27.378028    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:35.378010147 +0000 UTC m=+88.340309685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:27.378237 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:27.378043    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:27.378237 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:27.378076    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:35.37806574 +0000 UTC m=+88.340365292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:28.887768 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:28.887735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:28.888184 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:28.887904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:28.888184 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:28.887936    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:28.888184 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:28.888003    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:36.887988358 +0000 UTC m=+89.850287944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : secret "router-metrics-certs-default" not found
Sep 17 15:40:28.888184 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:28.888018    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:36.888011398 +0000 UTC m=+89.850310921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:35.439531 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:35.439388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:35.439531 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:35.439472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:35.439992 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:35.439597    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:35.439992 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:35.439691    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls podName:a9c30b22-cb0b-40ea-a35a-a45ebaecd115 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:51.439670861 +0000 UTC m=+104.401970410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wvvp2" (UID: "a9c30b22-cb0b-40ea-a35a-a45ebaecd115") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:35.439992 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:35.439946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:36.950036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:36.949999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:36.950478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:36.950048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:36.950478 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:36.950150    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:36.950478 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:36.950184    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:52.95016349 +0000 UTC m=+105.912463014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:36.950478 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:36.950208    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs podName:a6726151-1a0b-4092-a4f3-5356c7d72b22 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:52.950200813 +0000 UTC m=+105.912500337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs") pod "router-default-578cf489b4-4hnf6" (UID: "a6726151-1a0b-4092-a4f3-5356c7d72b22") : secret "router-metrics-certs-default" not found
Sep 17 15:40:44.712482 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.712448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:40:44.712962 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.712498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:40:44.714980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.714941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81-metrics-tls\") pod \"dns-default-4zz5s\" (UID: \"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81\") " pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:40:44.715414 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.715395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"image-registry-5dcff994bc-brfvh\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") " pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:40:44.744308 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.744286    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wq7zh\""
Sep 17 15:40:44.750867 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.750853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:40:44.766610 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.766584    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxcdv\""
Sep 17 15:40:44.773958 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:44.773927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:40:45.029959 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.029936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcff994bc-brfvh"]
Sep 17 15:40:45.032375 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:45.032351    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0a7c8e_f752_455c_95c0_94426646eb76.slice/crio-4ba90a45e079a1903c5084eafd76365c5527aa0b7b3144b9834e060908d7bc50 WatchSource:0}: Error finding container 4ba90a45e079a1903c5084eafd76365c5527aa0b7b3144b9834e060908d7bc50: Status 404 returned error can't find the container with id 4ba90a45e079a1903c5084eafd76365c5527aa0b7b3144b9834e060908d7bc50
Sep 17 15:40:45.049677 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.049654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4zz5s"]
Sep 17 15:40:45.073290 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:45.073269    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d73d927_bcfe_4ed9_9b5d_91e8d86e0d81.slice/crio-f4d73df200d1f306f92f56b158b3fab2dd8698498d64908a5c77b8df7e53317c WatchSource:0}: Error finding container f4d73df200d1f306f92f56b158b3fab2dd8698498d64908a5c77b8df7e53317c: Status 404 returned error can't find the container with id f4d73df200d1f306f92f56b158b3fab2dd8698498d64908a5c77b8df7e53317c
Sep 17 15:40:45.903469 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.903428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4zz5s" event={"ID":"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81","Type":"ContainerStarted","Data":"f4d73df200d1f306f92f56b158b3fab2dd8698498d64908a5c77b8df7e53317c"}
Sep 17 15:40:45.906565 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.905876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" event={"ID":"fa0a7c8e-f752-455c-95c0-94426646eb76","Type":"ContainerStarted","Data":"575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97"}
Sep 17 15:40:45.906565 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.905924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" event={"ID":"fa0a7c8e-f752-455c-95c0-94426646eb76","Type":"ContainerStarted","Data":"4ba90a45e079a1903c5084eafd76365c5527aa0b7b3144b9834e060908d7bc50"}
Sep 17 15:40:45.906565 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.906514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:40:45.933478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:45.933433    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" podStartSLOduration=97.933416228 podStartE2EDuration="1m37.933416228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:45.932120478 +0000 UTC m=+98.894420044" watchObservedRunningTime="2026-09-17 15:40:45.933416228 +0000 UTC m=+98.895715775"
Sep 17 15:40:46.911547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:46.911506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4zz5s" event={"ID":"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81","Type":"ContainerStarted","Data":"df137f98c56141e35a2ee7752faab13e0ad55f917b8aa248aa1369122772604d"}
Sep 17 15:40:46.911943 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:46.911608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4zz5s" event={"ID":"0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81","Type":"ContainerStarted","Data":"cb235f51cc3742df8dd01ff4086e9cd86da9f2c5839e4a80cbdb01628ae82ff1"}
Sep 17 15:40:46.931221 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:46.931168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4zz5s" podStartSLOduration=65.531294367 podStartE2EDuration="1m6.931152584s" totalImagesPullingTime="1.399858217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:46.929950995 +0000 UTC m=+99.892250554" watchObservedRunningTime="2026-09-17 15:40:46.931152584 +0000 UTC m=+99.893452130"
Sep 17 15:40:47.914223 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:47.914184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:40:49.840201 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:49.840165    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xpcqf"
Sep 17 15:40:50.059597 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.059552    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"]
Sep 17 15:40:50.091772 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.091716    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-9b8dh"]
Sep 17 15:40:50.091894 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.091829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.094495 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.094472    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 15:40:50.094610 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.094492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 15:40:50.094610 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.094561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hw2c4\""
Sep 17 15:40:50.105442 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.105424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"]
Sep 17 15:40:50.105534 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.105451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-9b8dh"]
Sep 17 15:40:50.105534 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.105508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:40:50.107860 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.107846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kvxg4\""
Sep 17 15:40:50.108168 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.108154    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:40:50.108209 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.108172    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:40:50.155816 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.155796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d605099e-24e9-49cb-a936-bc6fc847d771-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-t78tf\" (UID: \"d605099e-24e9-49cb-a936-bc6fc847d771\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.155914 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.155833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d605099e-24e9-49cb-a936-bc6fc847d771-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-t78tf\" (UID: \"d605099e-24e9-49cb-a936-bc6fc847d771\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.256320 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.256294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d605099e-24e9-49cb-a936-bc6fc847d771-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-t78tf\" (UID: \"d605099e-24e9-49cb-a936-bc6fc847d771\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.256409 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.256344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d605099e-24e9-49cb-a936-bc6fc847d771-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-t78tf\" (UID: \"d605099e-24e9-49cb-a936-bc6fc847d771\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.256409 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.256394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxl9\" (UniqueName: \"kubernetes.io/projected/72288695-a976-4e8e-886f-6b8924db3cf4-kube-api-access-zhxl9\") pod \"downloads-76f99584f4-9b8dh\" (UID: \"72288695-a976-4e8e-886f-6b8924db3cf4\") " pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:40:50.257002 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.256982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d605099e-24e9-49cb-a936-bc6fc847d771-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-t78tf\" (UID: \"d605099e-24e9-49cb-a936-bc6fc847d771\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.258557 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.258538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d605099e-24e9-49cb-a936-bc6fc847d771-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-t78tf\" (UID: \"d605099e-24e9-49cb-a936-bc6fc847d771\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.356907 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.356848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxl9\" (UniqueName: \"kubernetes.io/projected/72288695-a976-4e8e-886f-6b8924db3cf4-kube-api-access-zhxl9\") pod \"downloads-76f99584f4-9b8dh\" (UID: \"72288695-a976-4e8e-886f-6b8924db3cf4\") " pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:40:50.370446 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.370419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxl9\" (UniqueName: \"kubernetes.io/projected/72288695-a976-4e8e-886f-6b8924db3cf4-kube-api-access-zhxl9\") pod \"downloads-76f99584f4-9b8dh\" (UID: \"72288695-a976-4e8e-886f-6b8924db3cf4\") " pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:40:50.400065 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.400041    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"]
Sep 17 15:40:50.401225 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.401208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"
Sep 17 15:40:50.413323 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.413299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:40:50.425397 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.425381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:50.427822 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.427798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"]
Sep 17 15:40:50.428589 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.428573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 15:40:50.428670 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.428595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xmc5m\""
Sep 17 15:40:50.559333 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.559305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d981a759-2447-448d-9758-ee3f05f973b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-chq5w\" (UID: \"d981a759-2447-448d-9758-ee3f05f973b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:50.660251 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.660222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d981a759-2447-448d-9758-ee3f05f973b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-chq5w\" (UID: \"d981a759-2447-448d-9758-ee3f05f973b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:50.662353 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.662337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d981a759-2447-448d-9758-ee3f05f973b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-chq5w\" (UID: \"d981a759-2447-448d-9758-ee3f05f973b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:50.704027 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.704004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf"]
Sep 17 15:40:50.706915 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:50.706893    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd605099e_24e9_49cb_a936_bc6fc847d771.slice/crio-aa6d6e79d1e291b592ce5683476d3affd812bcceca12e2392735b0836d686508 WatchSource:0}: Error finding container aa6d6e79d1e291b592ce5683476d3affd812bcceca12e2392735b0836d686508: Status 404 returned error can't find the container with id aa6d6e79d1e291b592ce5683476d3affd812bcceca12e2392735b0836d686508
Sep 17 15:40:50.722492 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.722474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-9b8dh"]
Sep 17 15:40:50.724427 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:50.724405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72288695_a976_4e8e_886f_6b8924db3cf4.slice/crio-cda77f8375fed5b4c57c75240340c4530f95959e7adc4aa309faeced67764043 WatchSource:0}: Error finding container cda77f8375fed5b4c57c75240340c4530f95959e7adc4aa309faeced67764043: Status 404 returned error can't find the container with id cda77f8375fed5b4c57c75240340c4530f95959e7adc4aa309faeced67764043
Sep 17 15:40:50.733909 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.733894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:50.925435 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.925366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf" event={"ID":"d605099e-24e9-49cb-a936-bc6fc847d771","Type":"ContainerStarted","Data":"aa6d6e79d1e291b592ce5683476d3affd812bcceca12e2392735b0836d686508"}
Sep 17 15:40:50.926508 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:50.926482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-9b8dh" event={"ID":"72288695-a976-4e8e-886f-6b8924db3cf4","Type":"ContainerStarted","Data":"cda77f8375fed5b4c57c75240340c4530f95959e7adc4aa309faeced67764043"}
Sep 17 15:40:51.030994 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:51.030972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"]
Sep 17 15:40:51.033600 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:51.033578    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd981a759_2447_448d_9758_ee3f05f973b4.slice/crio-83c350f247c9904e6dc22c62401a42cf33f0296397c5f5c053225a0b3531249c WatchSource:0}: Error finding container 83c350f247c9904e6dc22c62401a42cf33f0296397c5f5c053225a0b3531249c: Status 404 returned error can't find the container with id 83c350f247c9904e6dc22c62401a42cf33f0296397c5f5c053225a0b3531249c
Sep 17 15:40:51.466751 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:51.466727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:51.469089 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:51.469064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9c30b22-cb0b-40ea-a35a-a45ebaecd115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvvp2\" (UID: \"a9c30b22-cb0b-40ea-a35a-a45ebaecd115\") " pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:51.736763 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:51.736682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvvp2"
Sep 17 15:40:51.932799 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:51.932744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w" event={"ID":"d981a759-2447-448d-9758-ee3f05f973b4","Type":"ContainerStarted","Data":"83c350f247c9904e6dc22c62401a42cf33f0296397c5f5c053225a0b3531249c"}
Sep 17 15:40:52.038908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.038828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvvp2"]
Sep 17 15:40:52.242820 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:52.242786    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c30b22_cb0b_40ea_a35a_a45ebaecd115.slice/crio-da5028bacffb0e7fbf3f069027bbd96c2b5ad2e87dfef47e6ba16cf929db0234 WatchSource:0}: Error finding container da5028bacffb0e7fbf3f069027bbd96c2b5ad2e87dfef47e6ba16cf929db0234: Status 404 returned error can't find the container with id da5028bacffb0e7fbf3f069027bbd96c2b5ad2e87dfef47e6ba16cf929db0234
Sep 17 15:40:52.940952 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.940654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf" event={"ID":"d605099e-24e9-49cb-a936-bc6fc847d771","Type":"ContainerStarted","Data":"6c9fb016028ad96606bfeef3fb23e4a8e6be7fd27e97e48d7e359cd716e54123"}
Sep 17 15:40:52.942546 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.942473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvp2" event={"ID":"a9c30b22-cb0b-40ea-a35a-a45ebaecd115","Type":"ContainerStarted","Data":"438fe2e12fd27f13921c9c0c372fe850bbe1e055faac99ef104acb1a3ba82153"}
Sep 17 15:40:52.942546 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.942522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvp2" event={"ID":"a9c30b22-cb0b-40ea-a35a-a45ebaecd115","Type":"ContainerStarted","Data":"da5028bacffb0e7fbf3f069027bbd96c2b5ad2e87dfef47e6ba16cf929db0234"}
Sep 17 15:40:52.944128 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.944098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w" event={"ID":"d981a759-2447-448d-9758-ee3f05f973b4","Type":"ContainerStarted","Data":"a331ace5ce42d543c9ca54716fc5b5e0b462b315bc443de21cf326751df3e953"}
Sep 17 15:40:52.944277 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.944259    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:52.949764 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.949747    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w"
Sep 17 15:40:52.960458 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.960413    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-t78tf" podStartSLOduration=0.852991664 podStartE2EDuration="2.960395165s" totalImagesPullingTime="2.107403501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:52.959376182 +0000 UTC m=+105.921675740" watchObservedRunningTime="2026-09-17 15:40:52.960395165 +0000 UTC m=+105.922694714"
Sep 17 15:40:52.976980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.976937    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-chq5w" podStartSLOduration=1.195010555 podStartE2EDuration="2.976924315s" totalImagesPullingTime="1.78191376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:52.976421399 +0000 UTC m=+105.938720968" watchObservedRunningTime="2026-09-17 15:40:52.976924315 +0000 UTC m=+105.939223852"
Sep 17 15:40:52.982222 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.982198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:52.982346 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.982331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:52.983007 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.982980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6726151-1a0b-4092-a4f3-5356c7d72b22-service-ca-bundle\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:52.985222 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:52.985198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6726151-1a0b-4092-a4f3-5356c7d72b22-metrics-certs\") pod \"router-default-578cf489b4-4hnf6\" (UID: \"a6726151-1a0b-4092-a4f3-5356c7d72b22\") " pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:53.246052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.246020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:53.477314 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.477290    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"]
Sep 17 15:40:53.479838 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.479821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.484135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.483789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:40:53.484135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.483818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 15:40:53.484135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.483826    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 15:40:53.484135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.483879    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ch9pw\""
Sep 17 15:40:53.484135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.483942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:40:53.484428 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.484159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:40:53.503174 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.503119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"]
Sep 17 15:40:53.538940 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.538893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-578cf489b4-4hnf6"]
Sep 17 15:40:53.541465 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:53.541444    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6726151_1a0b_4092_a4f3_5356c7d72b22.slice/crio-3eeea3691c2a632bf776fa5f3c82176eeb6ffa1ee80a1d19bf46f4506fe13537 WatchSource:0}: Error finding container 3eeea3691c2a632bf776fa5f3c82176eeb6ffa1ee80a1d19bf46f4506fe13537: Status 404 returned error can't find the container with id 3eeea3691c2a632bf776fa5f3c82176eeb6ffa1ee80a1d19bf46f4506fe13537
Sep 17 15:40:53.587735 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.587709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9561363d-ea2b-4f0f-b0d8-808683890341-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.587847 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.587750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9561363d-ea2b-4f0f-b0d8-808683890341-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.587847 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.587832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg42\" (UniqueName: \"kubernetes.io/projected/9561363d-ea2b-4f0f-b0d8-808683890341-kube-api-access-5pg42\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.587969 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.587877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561363d-ea2b-4f0f-b0d8-808683890341-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.688806 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.688772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561363d-ea2b-4f0f-b0d8-808683890341-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.688963 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.688868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9561363d-ea2b-4f0f-b0d8-808683890341-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.688963 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.688896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9561363d-ea2b-4f0f-b0d8-808683890341-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.689086 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.688962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg42\" (UniqueName: \"kubernetes.io/projected/9561363d-ea2b-4f0f-b0d8-808683890341-kube-api-access-5pg42\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.689611 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.689563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561363d-ea2b-4f0f-b0d8-808683890341-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.691688 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.691663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9561363d-ea2b-4f0f-b0d8-808683890341-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.691788 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.691738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9561363d-ea2b-4f0f-b0d8-808683890341-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.699108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.699085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg42\" (UniqueName: \"kubernetes.io/projected/9561363d-ea2b-4f0f-b0d8-808683890341-kube-api-access-5pg42\") pod \"prometheus-operator-68bcd7f987-k4cfz\" (UID: \"9561363d-ea2b-4f0f-b0d8-808683890341\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.791752 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.791692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"
Sep 17 15:40:53.950949 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.950760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-578cf489b4-4hnf6" event={"ID":"a6726151-1a0b-4092-a4f3-5356c7d72b22","Type":"ContainerStarted","Data":"eeb8ea9982887f297dbe0a4ebdbd207d1e67c4f38def72558b50f90774720e05"}
Sep 17 15:40:53.950949 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.950827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-578cf489b4-4hnf6" event={"ID":"a6726151-1a0b-4092-a4f3-5356c7d72b22","Type":"ContainerStarted","Data":"3eeea3691c2a632bf776fa5f3c82176eeb6ffa1ee80a1d19bf46f4506fe13537"}
Sep 17 15:40:53.954042 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.953999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvp2" event={"ID":"a9c30b22-cb0b-40ea-a35a-a45ebaecd115","Type":"ContainerStarted","Data":"c688bc35e5ba631a34ec24600ec5ce45b60085ee5fe2618c9ebf16da5df85c23"}
Sep 17 15:40:53.976142 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:53.976080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-578cf489b4-4hnf6" podStartSLOduration=32.976060192 podStartE2EDuration="32.976060192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:53.974538646 +0000 UTC m=+106.936838216" watchObservedRunningTime="2026-09-17 15:40:53.976060192 +0000 UTC m=+106.938359741"
Sep 17 15:40:54.073371 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.073260    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz"]
Sep 17 15:40:54.078406 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:54.078361    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9561363d_ea2b_4f0f_b0d8_808683890341.slice/crio-2cb57714897872d0c28caccac57e9f057d91e13aeb9183a1d14199b9acb20192 WatchSource:0}: Error finding container 2cb57714897872d0c28caccac57e9f057d91e13aeb9183a1d14199b9acb20192: Status 404 returned error can't find the container with id 2cb57714897872d0c28caccac57e9f057d91e13aeb9183a1d14199b9acb20192
Sep 17 15:40:54.246630 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.246601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:54.249309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.249284    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:54.966047 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.966013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvvp2" event={"ID":"a9c30b22-cb0b-40ea-a35a-a45ebaecd115","Type":"ContainerStarted","Data":"a055112d516a446414adeeeaacaf9ec9c441d1cdbe05436642158a802aeae0fb"}
Sep 17 15:40:54.967811 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.967784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz" event={"ID":"9561363d-ea2b-4f0f-b0d8-808683890341","Type":"ContainerStarted","Data":"2cb57714897872d0c28caccac57e9f057d91e13aeb9183a1d14199b9acb20192"}
Sep 17 15:40:54.968185 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.968166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:54.969460 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.969438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-578cf489b4-4hnf6"
Sep 17 15:40:54.988769 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:54.988728    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wvvp2" podStartSLOduration=34.189725178 podStartE2EDuration="35.988711959s" totalImagesPullingTime="1.798986781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:54.987072714 +0000 UTC m=+107.949372287" watchObservedRunningTime="2026-09-17 15:40:54.988711959 +0000 UTC m=+107.951011506"
Sep 17 15:40:55.973103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:55.973065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz" event={"ID":"9561363d-ea2b-4f0f-b0d8-808683890341","Type":"ContainerStarted","Data":"4f8e915d5b14543ad87d37dbe86161da0ff3c9c19966057305eb70a4de31b8e1"}
Sep 17 15:40:55.973555 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:55.973136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz" event={"ID":"9561363d-ea2b-4f0f-b0d8-808683890341","Type":"ContainerStarted","Data":"897409caa69d7643cbe6ee57b136bd3e80635a33068fdf2e53a7e9b642c87b69"}
Sep 17 15:40:55.993933 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:55.993878    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-k4cfz" podStartSLOduration=1.5322737979999999 podStartE2EDuration="2.993834161s" totalImagesPullingTime="1.461560363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:55.993610865 +0000 UTC m=+108.955910433" watchObservedRunningTime="2026-09-17 15:40:55.993834161 +0000 UTC m=+108.956133707"
Sep 17 15:40:57.869779 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.869745    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"]
Sep 17 15:40:57.872300 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.872274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:57.874688 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.874619    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 15:40:57.879299 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.875393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:40:57.879299 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.875465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sl6v5\""
Sep 17 15:40:57.881617 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.881566    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"]
Sep 17 15:40:57.884003 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.883985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:57.886803 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.886770    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:40:57.886895 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.886841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mncz2\""
Sep 17 15:40:57.887875 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.887856    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 15:40:57.888138 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.887929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 15:40:57.889509 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.889490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"]
Sep 17 15:40:57.898525 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.898504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cxb98"]
Sep 17 15:40:57.900909 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.900891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"]
Sep 17 15:40:57.901026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.901007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:57.903072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.903055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:40:57.903442 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.903425    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:40:57.903559 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.903518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8hqsc\""
Sep 17 15:40:57.903937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.903919    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:40:57.920222 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:57.920200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4zz5s"
Sep 17 15:40:58.022699 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17469123-740f-4a83-a769-c4e98ce51c2b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.022835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.022835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17469123-740f-4a83-a769-c4e98ce51c2b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.022835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb7r\" (UniqueName: \"kubernetes.io/projected/17469123-740f-4a83-a769-c4e98ce51c2b-kube-api-access-lxb7r\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.022835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.022835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-wtmp\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-root\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-tls\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.022980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6ck2\" (UniqueName: \"kubernetes.io/projected/858612e0-c5da-489e-a20d-c53433a8a538-kube-api-access-m6ck2\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-kube-api-access-rpnkg\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-textfile\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-sys\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.023103 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.023620 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.023620 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.023133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/858612e0-c5da-489e-a20d-c53433a8a538-metrics-client-ca\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124406 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17469123-740f-4a83-a769-c4e98ce51c2b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.124547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17469123-740f-4a83-a769-c4e98ce51c2b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.124547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb7r\" (UniqueName: \"kubernetes.io/projected/17469123-740f-4a83-a769-c4e98ce51c2b-kube-api-access-lxb7r\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.124547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.124547 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-wtmp\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-root\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-tls\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ck2\" (UniqueName: \"kubernetes.io/projected/858612e0-c5da-489e-a20d-c53433a8a538-kube-api-access-m6ck2\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-kube-api-access-rpnkg\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17469123-740f-4a83-a769-c4e98ce51c2b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.124804 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-textfile\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-sys\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.124889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/858612e0-c5da-489e-a20d-c53433a8a538-metrics-client-ca\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-textfile\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:58.125153    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:58.125220    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-tls podName:1c619fea-0a0d-4b6b-b6e1-771811e8fc60 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:58.625201656 +0000 UTC m=+111.587501183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-2jc4j" (UID: "1c619fea-0a0d-4b6b-b6e1-771811e8fc60") : secret "openshift-state-metrics-tls" not found
Sep 17 15:40:58.125282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17469123-740f-4a83-a769-c4e98ce51c2b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.125723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-root\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-wtmp\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/858612e0-c5da-489e-a20d-c53433a8a538-metrics-client-ca\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.125976 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.125976 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.125976 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.125952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/858612e0-c5da-489e-a20d-c53433a8a538-sys\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.128113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.128071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.128343 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.128311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.128735 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.128697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/858612e0-c5da-489e-a20d-c53433a8a538-node-exporter-tls\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.128827 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.128749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.128934 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.128909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17469123-740f-4a83-a769-c4e98ce51c2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.134691 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.134667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb7r\" (UniqueName: \"kubernetes.io/projected/17469123-740f-4a83-a769-c4e98ce51c2b-kube-api-access-lxb7r\") pod \"kube-state-metrics-59fbdd77d-pnsn6\" (UID: \"17469123-740f-4a83-a769-c4e98ce51c2b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.134864 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.134840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-kube-api-access-rpnkg\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.135468 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.135450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ck2\" (UniqueName: \"kubernetes.io/projected/858612e0-c5da-489e-a20d-c53433a8a538-kube-api-access-m6ck2\") pod \"node-exporter-cxb98\" (UID: \"858612e0-c5da-489e-a20d-c53433a8a538\") " pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.194966 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.194945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"
Sep 17 15:40:58.211775 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.211754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxb98"
Sep 17 15:40:58.222948 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:58.222923    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858612e0_c5da_489e_a20d_c53433a8a538.slice/crio-87746a43371b4a4ec0641259f7fe762c8896190e508ae8a652bf531821ff8d85 WatchSource:0}: Error finding container 87746a43371b4a4ec0641259f7fe762c8896190e508ae8a652bf531821ff8d85: Status 404 returned error can't find the container with id 87746a43371b4a4ec0641259f7fe762c8896190e508ae8a652bf531821ff8d85
Sep 17 15:40:58.474470 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.474309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6"]
Sep 17 15:40:58.476860 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:58.476828    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17469123_740f_4a83_a769_c4e98ce51c2b.slice/crio-447a7717a5534c78516c43665107e4bb3c0549d3a21fd7aa06520322d75f838f WatchSource:0}: Error finding container 447a7717a5534c78516c43665107e4bb3c0549d3a21fd7aa06520322d75f838f: Status 404 returned error can't find the container with id 447a7717a5534c78516c43665107e4bb3c0549d3a21fd7aa06520322d75f838f
Sep 17 15:40:58.629146 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.629114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.631996 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.631966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c619fea-0a0d-4b6b-b6e1-771811e8fc60-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2jc4j\" (UID: \"1c619fea-0a0d-4b6b-b6e1-771811e8fc60\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.783907 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.783841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"
Sep 17 15:40:58.989674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.989633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6" event={"ID":"17469123-740f-4a83-a769-c4e98ce51c2b","Type":"ContainerStarted","Data":"447a7717a5534c78516c43665107e4bb3c0549d3a21fd7aa06520322d75f838f"}
Sep 17 15:40:58.991168 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:58.991139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxb98" event={"ID":"858612e0-c5da-489e-a20d-c53433a8a538","Type":"ContainerStarted","Data":"87746a43371b4a4ec0641259f7fe762c8896190e508ae8a652bf531821ff8d85"}
Sep 17 15:40:59.008570 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.008345    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:59.012149 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.012129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.015874 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.014884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:40:59.015874 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.015214    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zzm7n\""
Sep 17 15:40:59.015874 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.015436    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:40:59.015874 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.015632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:40:59.015874 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.015691    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:40:59.015874 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.015874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:40:59.016218 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.016139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:40:59.016427 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.016405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:40:59.016700 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.016647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:40:59.016867 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.016850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:40:59.028992 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.028967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:59.073274 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.072837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j"]
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-config-volume\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-web-config\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrx8z\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-kube-api-access-rrx8z\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-config-out\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.133849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.133622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234565 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-config-out\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.234755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-config-volume\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-web-config\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrx8z\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-kube-api-access-rrx8z\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235072 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.234962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.235412 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:59.235302    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 15:40:59.235412 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:40:59.235368    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls podName:f377e2ee-3090-4835-b030-74c98b295644 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:59.735348414 +0000 UTC m=+112.697647942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f377e2ee-3090-4835-b030-74c98b295644") : secret "alertmanager-main-tls" not found
Sep 17 15:40:59.236921 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.236656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.238679 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.237542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.239790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.239695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.241888 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.241862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.241982 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.241964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-web-config\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.242372 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.242322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-config-volume\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.243090 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.243043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.244181 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.243974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.244181 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.244155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-config-out\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.249086 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.249062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.253649 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.253481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrx8z\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-kube-api-access-rrx8z\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.305005 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:40:59.304973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c619fea_0a0d_4b6b_b6e1_771811e8fc60.slice/crio-9b66e7a43c2709df5ba171d491ce578cc9d6768bd4e909261d95838d03cd2410 WatchSource:0}: Error finding container 9b66e7a43c2709df5ba171d491ce578cc9d6768bd4e909261d95838d03cd2410: Status 404 returned error can't find the container with id 9b66e7a43c2709df5ba171d491ce578cc9d6768bd4e909261d95838d03cd2410
Sep 17 15:40:59.738905 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.738868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.741981 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.741954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.926944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.926513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:59.930608 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.930575    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"]
Sep 17 15:40:59.935276 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.934979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:40:59.938235 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.938212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 15:40:59.938357 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.938212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 15:40:59.938596 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.938495    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 15:40:59.938995 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.938977    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 15:40:59.939300 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.939285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 15:40:59.939495 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.939338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fbv66\""
Sep 17 15:40:59.939603 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.939338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f3vna0br9hjh1\""
Sep 17 15:40:59.956659 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:40:59.956615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"]
Sep 17 15:41:00.004044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.003708    2563 generic.go:356] "Generic (PLEG): container finished" podID="858612e0-c5da-489e-a20d-c53433a8a538" containerID="53696e3cb2c3a0394fa671285909ffe1a93f43e80db9d1f80b669a6cb28756f1" exitCode=0
Sep 17 15:41:00.004044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.003876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxb98" event={"ID":"858612e0-c5da-489e-a20d-c53433a8a538","Type":"ContainerDied","Data":"53696e3cb2c3a0394fa671285909ffe1a93f43e80db9d1f80b669a6cb28756f1"}
Sep 17 15:41:00.012516 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.012475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j" event={"ID":"1c619fea-0a0d-4b6b-b6e1-771811e8fc60","Type":"ContainerStarted","Data":"cd2de54a8b6d4443fc75947d9f2e01ea8f1d08125e6ad28597274850105398e3"}
Sep 17 15:41:00.012628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.012549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j" event={"ID":"1c619fea-0a0d-4b6b-b6e1-771811e8fc60","Type":"ContainerStarted","Data":"651b42df661fe9b925221baa3a11aadf7735513d207ebf797513ab7c18cc01c1"}
Sep 17 15:41:00.012628 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.012586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j" event={"ID":"1c619fea-0a0d-4b6b-b6e1-771811e8fc60","Type":"ContainerStarted","Data":"9b66e7a43c2709df5ba171d491ce578cc9d6768bd4e909261d95838d03cd2410"}
Sep 17 15:41:00.016659 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.016427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6" event={"ID":"17469123-740f-4a83-a769-c4e98ce51c2b","Type":"ContainerStarted","Data":"f789e369d03984e7abae0716f9e66a0819365a953d93cf5ef495656d943f4fd5"}
Sep 17 15:41:00.016659 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.016482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6" event={"ID":"17469123-740f-4a83-a769-c4e98ce51c2b","Type":"ContainerStarted","Data":"3783566f4463b69600e3e3cb4658f2f23ad4f902cacec66cf456bc71ca34c9ad"}
Sep 17 15:41:00.016659 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.016511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6" event={"ID":"17469123-740f-4a83-a769-c4e98ce51c2b","Type":"ContainerStarted","Data":"3d9cbb1ca913f845ccdcbe3903c754300720903ab3a2a6098873141db9a70c41"}
Sep 17 15:41:00.040496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-metrics-client-ca\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040720 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-tls\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040772 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mb64\" (UniqueName: \"kubernetes.io/projected/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-kube-api-access-4mb64\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040772 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.040867 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.040849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-grpc-tls\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.047438 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.047394    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pnsn6" podStartSLOduration=1.788390461 podStartE2EDuration="3.047379645s" totalImagesPullingTime="1.258989184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:00.0462635 +0000 UTC m=+113.008563068" watchObservedRunningTime="2026-09-17 15:41:00.047379645 +0000 UTC m=+113.009679188"
Sep 17 15:41:00.141446 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mb64\" (UniqueName: \"kubernetes.io/projected/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-kube-api-access-4mb64\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.141573 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.141691 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-grpc-tls\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.141798 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.141798 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-metrics-client-ca\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.141908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.141908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.142016 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.141989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-tls\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.144229 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.143991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-metrics-client-ca\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.146609 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.146583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.147232 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.147207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-grpc-tls\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.148197 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.148137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-tls\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.148611 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.148432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.148611 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.148585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.150008 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.149984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.151836 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.151818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mb64\" (UniqueName: \"kubernetes.io/projected/a37d3c8b-1898-41bd-ad05-3b8d6b1075a4-kube-api-access-4mb64\") pod \"thanos-querier-5c6c68f5c-rppcb\" (UID: \"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4\") " pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.225682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.225608    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:41:00.230565 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:41:00.230536    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf377e2ee_3090_4835_b030_74c98b295644.slice/crio-b310809293b683a7afed715e1763f8d5e3af8c0960526d8fbbf61f01e22ce862 WatchSource:0}: Error finding container b310809293b683a7afed715e1763f8d5e3af8c0960526d8fbbf61f01e22ce862: Status 404 returned error can't find the container with id b310809293b683a7afed715e1763f8d5e3af8c0960526d8fbbf61f01e22ce862
Sep 17 15:41:00.250050 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.249857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:00.547676 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:00.547594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"]
Sep 17 15:41:01.023251 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.023215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxb98" event={"ID":"858612e0-c5da-489e-a20d-c53433a8a538","Type":"ContainerStarted","Data":"d9d0f18886e44a4e60c0c6e28fc59c8bee8b915137c58033ada7e5e0de06f5d1"}
Sep 17 15:41:01.023720 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.023304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxb98" event={"ID":"858612e0-c5da-489e-a20d-c53433a8a538","Type":"ContainerStarted","Data":"30fa6944d1d8edc92ee8c693f03eb859fb35dfddbcfaac88387c78b934350171"}
Sep 17 15:41:01.024619 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.024595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"b310809293b683a7afed715e1763f8d5e3af8c0960526d8fbbf61f01e22ce862"}
Sep 17 15:41:01.048015 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.047964    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cxb98" podStartSLOduration=2.478642238 podStartE2EDuration="4.047945703s" totalImagesPullingTime="1.482948712s" totalInitContainerRuntime="86.354753ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:01.045905372 +0000 UTC m=+114.008204936" watchObservedRunningTime="2026-09-17 15:41:01.047945703 +0000 UTC m=+114.010245252"
Sep 17 15:41:01.824603 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.823995    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78cf84dc48-nrs9r"]
Sep 17 15:41:01.826646 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.826615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:01.833872 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.833849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:41:01.834432 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.833893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:41:01.834432 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.833908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:41:01.835015 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.834995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:41:01.835154 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.835015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lrjhd\""
Sep 17 15:41:01.835531 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.835417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:41:01.842776 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.842756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78cf84dc48-nrs9r"]
Sep 17 15:41:01.959895 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.959860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-oauth-config\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:01.960051 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.959911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-console-config\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:01.960051 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.959958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-serving-cert\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:01.960051 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.960026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-service-ca\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:01.960177 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.960089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-oauth-serving-cert\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:01.960177 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:01.960109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpxw\" (UniqueName: \"kubernetes.io/projected/31867d5c-95a6-4e40-b056-d90b98617819-kube-api-access-jjpxw\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.060938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.060888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-service-ca\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.061389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.060952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-oauth-serving-cert\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.061389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.060981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpxw\" (UniqueName: \"kubernetes.io/projected/31867d5c-95a6-4e40-b056-d90b98617819-kube-api-access-jjpxw\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.061389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.061030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-oauth-config\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.061389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.061068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-console-config\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.061389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.061101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-serving-cert\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.061849 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.061821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-service-ca\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.062026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.062000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-oauth-serving-cert\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.062315 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.062273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-console-config\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.064003 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.063948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-oauth-config\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.064178 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.064136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-serving-cert\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.071254 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.071218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpxw\" (UniqueName: \"kubernetes.io/projected/31867d5c-95a6-4e40-b056-d90b98617819-kube-api-access-jjpxw\") pod \"console-78cf84dc48-nrs9r\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") " pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.137986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.137962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:02.635563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.635533    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"]
Sep 17 15:41:02.637931 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.637911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.640309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.640262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 15:41:02.640309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.640276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h8xjg\""
Sep 17 15:41:02.648471 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.648390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"]
Sep 17 15:41:02.767459 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.767424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9aba322d-16d9-4df4-968a-7878d9748414-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4pxms\" (UID: \"9aba322d-16d9-4df4-968a-7878d9748414\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.767459 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.767465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55c9x\" (UniqueName: \"kubernetes.io/projected/9aba322d-16d9-4df4-968a-7878d9748414-kube-api-access-55c9x\") pod \"monitoring-plugin-5d85778b9d-4pxms\" (UID: \"9aba322d-16d9-4df4-968a-7878d9748414\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.868754 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.868717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9aba322d-16d9-4df4-968a-7878d9748414-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4pxms\" (UID: \"9aba322d-16d9-4df4-968a-7878d9748414\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.868754 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.868763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55c9x\" (UniqueName: \"kubernetes.io/projected/9aba322d-16d9-4df4-968a-7878d9748414-kube-api-access-55c9x\") pod \"monitoring-plugin-5d85778b9d-4pxms\" (UID: \"9aba322d-16d9-4df4-968a-7878d9748414\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.871790 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.871764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9aba322d-16d9-4df4-968a-7878d9748414-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4pxms\" (UID: \"9aba322d-16d9-4df4-968a-7878d9748414\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.882556 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.882499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55c9x\" (UniqueName: \"kubernetes.io/projected/9aba322d-16d9-4df4-968a-7878d9748414-kube-api-access-55c9x\") pod \"monitoring-plugin-5d85778b9d-4pxms\" (UID: \"9aba322d-16d9-4df4-968a-7878d9748414\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:02.949659 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:02.949576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:03.097951 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.096228    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55db5d8686-r5lpn"]
Sep 17 15:41:03.099194 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.099136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.102202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.102024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 15:41:03.102202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.102054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 15:41:03.102202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.102080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8jv5r\""
Sep 17 15:41:03.102202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.102104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 15:41:03.102202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.102106    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 15:41:03.102202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.102142    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 15:41:03.107253 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.107219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 15:41:03.113563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.113541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55db5d8686-r5lpn"]
Sep 17 15:41:03.273732 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-metrics-client-ca\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273732 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-secret-telemeter-client\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273732 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-serving-certs-ca-bundle\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-telemeter-client-tls\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-federate-client-tls\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqs65\" (UniqueName: \"kubernetes.io/projected/aa46cf83-2bab-47d5-a066-10c7c77474e6-kube-api-access-kqs65\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.273986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.273957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.374680 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-serving-certs-ca-bundle\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.374680 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-telemeter-client-tls\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.374886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-federate-client-tls\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.374886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqs65\" (UniqueName: \"kubernetes.io/projected/aa46cf83-2bab-47d5-a066-10c7c77474e6-kube-api-access-kqs65\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.374886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.374886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.375075 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-metrics-client-ca\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.375075 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.374990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-secret-telemeter-client\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.375716 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.375658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-serving-certs-ca-bundle\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.376005 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.375958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-metrics-client-ca\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.376112 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.376013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa46cf83-2bab-47d5-a066-10c7c77474e6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.377737 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.377699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-telemeter-client-tls\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.377909 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.377881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.377992 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.377887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-federate-client-tls\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.379200 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.379182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aa46cf83-2bab-47d5-a066-10c7c77474e6-secret-telemeter-client\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.390905 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.390877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqs65\" (UniqueName: \"kubernetes.io/projected/aa46cf83-2bab-47d5-a066-10c7c77474e6-kube-api-access-kqs65\") pod \"telemeter-client-55db5d8686-r5lpn\" (UID: \"aa46cf83-2bab-47d5-a066-10c7c77474e6\") " pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:03.411139 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:03.411115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn"
Sep 17 15:41:04.141130 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.141086    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:41:04.146834 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.146808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.150530 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.150505    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:41:04.151463 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.151319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:41:04.151463 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.151362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:41:04.151765 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.151745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:41:04.151920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.151747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:41:04.153108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.152604    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:41:04.153108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.152764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:41:04.153108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.152780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:41:04.153108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.152904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:41:04.153108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.152943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:41:04.153108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.153022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:41:04.153452 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.153325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:41:04.153503 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.153467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:41:04.153567 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.153550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82bolsmjilt5v\""
Sep 17 15:41:04.153810 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.153789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6znx\""
Sep 17 15:41:04.154217 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.154195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:41:04.154601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.154575    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:41:04.169646 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.169621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:41:04.285324 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285487 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285487 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285487 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285487 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285487 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.285758 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.286180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.286180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.286180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.286180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.286180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.285896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99pd\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-kube-api-access-w99pd\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387003 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.386968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387170 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387170 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387170 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387170 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387170 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.387441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388183 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.387665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388183 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388385 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388385 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388385 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388385 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388600 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388600 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388600 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388600 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388600 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388600 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99pd\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-kube-api-access-w99pd\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388895 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.388895 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.388678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.390513 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.390393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.390513 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.390436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.390684 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.390538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.390965 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.390936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.391550 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.391189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.391666 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.391629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.392840 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.392107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.392840 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.392154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.392840 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.392321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.393078 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.393048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.394387 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.394335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.395155 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.395131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.395454 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.395413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.395454 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.395440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.395591 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.395439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.395989 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.395965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.406387 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.406365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99pd\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-kube-api-access-w99pd\") pod \"prometheus-k8s-0\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.460642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.460451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:04.755961 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.755876    2563 patch_prober.go:28] interesting pod/image-registry-5dcff994bc-brfvh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:41:04.755961 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:04.755926    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" podUID="fa0a7c8e-f752-455c-95c0-94426646eb76" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:41:07.537763 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:41:07.537729    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37d3c8b_1898_41bd_ad05_3b8d6b1075a4.slice/crio-1d72aa827a74b5983d9789a5e0419f03de5bb099e6679d78badd874df239fcc4 WatchSource:0}: Error finding container 1d72aa827a74b5983d9789a5e0419f03de5bb099e6679d78badd874df239fcc4: Status 404 returned error can't find the container with id 1d72aa827a74b5983d9789a5e0419f03de5bb099e6679d78badd874df239fcc4
Sep 17 15:41:07.922038 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:07.921715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:41:08.056937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.056846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"1d72aa827a74b5983d9789a5e0419f03de5bb099e6679d78badd874df239fcc4"}
Sep 17 15:41:08.065999 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.065288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j" event={"ID":"1c619fea-0a0d-4b6b-b6e1-771811e8fc60","Type":"ContainerStarted","Data":"f0598e03b6c2863e5c8caaeb86c20d85bb2eb71ec9c057ce65a198bf95f6c21c"}
Sep 17 15:41:08.070734 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.070542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-9b8dh" event={"ID":"72288695-a976-4e8e-886f-6b8924db3cf4","Type":"ContainerStarted","Data":"74e4452622810a9206baf14240ce14f547d450294764e57ad7b4af36c00b5354"}
Sep 17 15:41:08.072037 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.071289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:41:08.074030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.072704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78cf84dc48-nrs9r"]
Sep 17 15:41:08.074030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.072953    2563 patch_prober.go:28] interesting pod/downloads-76f99584f4-9b8dh container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused" start-of-body=
Sep 17 15:41:08.074030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.072995    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-9b8dh" podUID="72288695-a976-4e8e-886f-6b8924db3cf4" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused"
Sep 17 15:41:08.076392 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:41:08.076347    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31867d5c_95a6_4e40_b056_d90b98617819.slice/crio-d76e61cc68d5c65a98f02bdd6c7f52125ef82a3bb8e147e210cd531042137fd8 WatchSource:0}: Error finding container d76e61cc68d5c65a98f02bdd6c7f52125ef82a3bb8e147e210cd531042137fd8: Status 404 returned error can't find the container with id d76e61cc68d5c65a98f02bdd6c7f52125ef82a3bb8e147e210cd531042137fd8
Sep 17 15:41:08.097324 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.096834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"]
Sep 17 15:41:08.098037 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.097995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2jc4j" podStartSLOduration=3.110184935 podStartE2EDuration="11.097978613s" totalImagesPullingTime="7.987793678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:08.090560759 +0000 UTC m=+121.052860322" watchObservedRunningTime="2026-09-17 15:41:08.097978613 +0000 UTC m=+121.060278162"
Sep 17 15:41:08.099977 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.098348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55db5d8686-r5lpn"]
Sep 17 15:41:08.103355 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:41:08.103335    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46cf83_2bab_47d5_a066_10c7c77474e6.slice/crio-c9f4da56aba8933dcc81829f896ed5b780d27ef38e4baec28902756831c60dce WatchSource:0}: Error finding container c9f4da56aba8933dcc81829f896ed5b780d27ef38e4baec28902756831c60dce: Status 404 returned error can't find the container with id c9f4da56aba8933dcc81829f896ed5b780d27ef38e4baec28902756831c60dce
Sep 17 15:41:08.103884 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.103863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:41:08.104359 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:41:08.104339    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28846b8_0e84_44ae_bd11_7e8ba96b279b.slice/crio-97c81d44cf4f828524c02390d5068d0c729c682ff3df8f26eb0947d9f85d3198 WatchSource:0}: Error finding container 97c81d44cf4f828524c02390d5068d0c729c682ff3df8f26eb0947d9f85d3198: Status 404 returned error can't find the container with id 97c81d44cf4f828524c02390d5068d0c729c682ff3df8f26eb0947d9f85d3198
Sep 17 15:41:08.113144 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:08.113100    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-9b8dh" podStartSLOduration=0.99431733 podStartE2EDuration="18.113082564s" totalImagesPullingTime="17.118765234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:08.11282576 +0000 UTC m=+121.075125330" watchObservedRunningTime="2026-09-17 15:41:08.113082564 +0000 UTC m=+121.075382112"
Sep 17 15:41:09.085222 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.085142    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272" exitCode=0
Sep 17 15:41:09.085665 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.085258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272"}
Sep 17 15:41:09.091338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.090990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cf84dc48-nrs9r" event={"ID":"31867d5c-95a6-4e40-b056-d90b98617819","Type":"ContainerStarted","Data":"d76e61cc68d5c65a98f02bdd6c7f52125ef82a3bb8e147e210cd531042137fd8"}
Sep 17 15:41:09.098799 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.098673    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b" exitCode=0
Sep 17 15:41:09.100277 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.100136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"}
Sep 17 15:41:09.100277 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.100206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"97c81d44cf4f828524c02390d5068d0c729c682ff3df8f26eb0947d9f85d3198"}
Sep 17 15:41:09.104425 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.104339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn" event={"ID":"aa46cf83-2bab-47d5-a066-10c7c77474e6","Type":"ContainerStarted","Data":"c9f4da56aba8933dcc81829f896ed5b780d27ef38e4baec28902756831c60dce"}
Sep 17 15:41:09.110213 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.109681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms" event={"ID":"9aba322d-16d9-4df4-968a-7878d9748414","Type":"ContainerStarted","Data":"0d89e0a87258a2a99ce45a1c151f56bd559ae757398b338d691752eec7ce422e"}
Sep 17 15:41:09.113178 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:09.113018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-9b8dh"
Sep 17 15:41:12.186426 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:12.186341    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dcff994bc-brfvh"]
Sep 17 15:41:17.176877 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.176752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn" event={"ID":"aa46cf83-2bab-47d5-a066-10c7c77474e6","Type":"ContainerStarted","Data":"0190a693c656f227ab4b6149d5225519938244410d7bca31b74f816386295370"}
Sep 17 15:41:17.181264 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.181212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms" event={"ID":"9aba322d-16d9-4df4-968a-7878d9748414","Type":"ContainerStarted","Data":"e5d0d8b90d9e226eca098c3c41ce88f0162499604cceef1efef80a0f68cd6b69"}
Sep 17 15:41:17.183650 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.183632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:17.183807 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.183792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cf84dc48-nrs9r" event={"ID":"31867d5c-95a6-4e40-b056-d90b98617819","Type":"ContainerStarted","Data":"89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247"}
Sep 17 15:41:17.189338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.189068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms"
Sep 17 15:41:17.204727 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.204599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4pxms" podStartSLOduration=6.364371395 podStartE2EDuration="15.204581905s" totalImagesPullingTime="8.84021051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:17.204052834 +0000 UTC m=+130.166352412" watchObservedRunningTime="2026-09-17 15:41:17.204581905 +0000 UTC m=+130.166881454"
Sep 17 15:41:17.224937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.224894    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78cf84dc48-nrs9r" podStartSLOduration=7.371247159 podStartE2EDuration="16.224879353s" totalImagesPullingTime="8.853632194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:17.222608069 +0000 UTC m=+130.184907635" watchObservedRunningTime="2026-09-17 15:41:17.224879353 +0000 UTC m=+130.187178898"
Sep 17 15:41:17.331087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.330499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:41:17.334670 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.334636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932cded3-c571-411f-b440-19ac1bd83639-metrics-certs\") pod \"network-metrics-daemon-stkm5\" (UID: \"932cded3-c571-411f-b440-19ac1bd83639\") " pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:41:17.344474 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.343649    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm2sj\""
Sep 17 15:41:17.352053 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.351226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stkm5"
Sep 17 15:41:17.703558 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:17.703532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stkm5"]
Sep 17 15:41:17.713470 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:41:17.707745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932cded3_c571_411f_b440_19ac1bd83639.slice/crio-641c9997e90e4fabb8f0c62b4312e5b65c27d3d98ab54d0a2f8402b05a9bd1e2 WatchSource:0}: Error finding container 641c9997e90e4fabb8f0c62b4312e5b65c27d3d98ab54d0a2f8402b05a9bd1e2: Status 404 returned error can't find the container with id 641c9997e90e4fabb8f0c62b4312e5b65c27d3d98ab54d0a2f8402b05a9bd1e2
Sep 17 15:41:18.196543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.196503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"040ad32290555da2608bb53d6d8080c2239f9c76d69fd4aebe75856187de0396"}
Sep 17 15:41:18.196980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.196641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"64f0a21efb4e99610089c88f00b32454c7037cf68593bc8391f014a5e7a7942a"}
Sep 17 15:41:18.196980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.196686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"3cfc7326fde83ccbfd90a7262e4ed0a7312e41668eaddad4e0b156beffad7ad4"}
Sep 17 15:41:18.200962 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.200927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"}
Sep 17 15:41:18.201113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.201011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"}
Sep 17 15:41:18.201113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.201059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"}
Sep 17 15:41:18.201113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.201098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"}
Sep 17 15:41:18.201286 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.201137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"}
Sep 17 15:41:18.201286 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.201179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerStarted","Data":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"}
Sep 17 15:41:18.203534 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.203504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn" event={"ID":"aa46cf83-2bab-47d5-a066-10c7c77474e6","Type":"ContainerStarted","Data":"20f98570255d9d28f505857a48de48f1bc0355e32f94eb5566a4e1269dc7abf5"}
Sep 17 15:41:18.203658 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.203569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn" event={"ID":"aa46cf83-2bab-47d5-a066-10c7c77474e6","Type":"ContainerStarted","Data":"caf49d217cec0c7888e225dd5b5b44cc4fddcb95ea35b4bb121e6e2453c8fb1e"}
Sep 17 15:41:18.204862 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.204820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stkm5" event={"ID":"932cded3-c571-411f-b440-19ac1bd83639","Type":"ContainerStarted","Data":"641c9997e90e4fabb8f0c62b4312e5b65c27d3d98ab54d0a2f8402b05a9bd1e2"}
Sep 17 15:41:18.208504 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.208479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c"}
Sep 17 15:41:18.208586 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.208551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559"}
Sep 17 15:41:18.208740 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.208593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254"}
Sep 17 15:41:18.208801 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.208787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa"}
Sep 17 15:41:18.208875 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.208859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3"}
Sep 17 15:41:18.233088 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.233046    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.776025092 podStartE2EDuration="14.233031051s" totalImagesPullingTime="8.433278819s" totalInitContainerRuntime="23.72714ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:41:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:18.231346541 +0000 UTC m=+131.193646113" watchObservedRunningTime="2026-09-17 15:41:18.233031051 +0000 UTC m=+131.195330599"
Sep 17 15:41:18.254429 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.254376    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55db5d8686-r5lpn" podStartSLOduration=6.60959992 podStartE2EDuration="15.254356114s" totalImagesPullingTime="8.644756194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:18.253182637 +0000 UTC m=+131.215482248" watchObservedRunningTime="2026-09-17 15:41:18.254356114 +0000 UTC m=+131.216655668"
Sep 17 15:41:18.302999 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:18.302964    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78cf84dc48-nrs9r"]
Sep 17 15:41:19.218116 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:19.217984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerStarted","Data":"2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58"}
Sep 17 15:41:19.221158 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:19.221128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"1f7223cb5a7a78fc9d68da0ceb2a6a3d23b4dd7d06c296e160cb873f5f781db9"}
Sep 17 15:41:19.250836 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:19.250789    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.722681418 podStartE2EDuration="21.250772761s" totalImagesPullingTime="17.500764824s" totalInitContainerRuntime="27.326519ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:19.249559625 +0000 UTC m=+132.211859192" watchObservedRunningTime="2026-09-17 15:41:19.250772761 +0000 UTC m=+132.213072309"
Sep 17 15:41:19.462261 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:19.462151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:20.229550 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.229515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"dab38e7181ccbce68433310e5193414d1d00966722fb53984244c4c94ed3365f"}
Sep 17 15:41:20.230104 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.229596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" event={"ID":"a37d3c8b-1898-41bd-ad05-3b8d6b1075a4","Type":"ContainerStarted","Data":"868faabab946823a0676d7cd0a68bb34810b98d6f3482cee6350ab0a0d962778"}
Sep 17 15:41:20.230104 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.229643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:20.231234 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.231201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stkm5" event={"ID":"932cded3-c571-411f-b440-19ac1bd83639","Type":"ContainerStarted","Data":"c574d20d5c9cadcd4218dc059396bfb0a7108dc8c47114640f238bc3a97668ac"}
Sep 17 15:41:20.231371 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.231272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stkm5" event={"ID":"932cded3-c571-411f-b440-19ac1bd83639","Type":"ContainerStarted","Data":"c4684f98068d09ba9bd79ce2d2e092f1cb1bdca8ab335fbec2b912b585f80b3c"}
Sep 17 15:41:20.266446 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.266399    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb" podStartSLOduration=10.485986827 podStartE2EDuration="21.266383568s" totalImagesPullingTime="10.780396741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:20.264289813 +0000 UTC m=+133.226589373" watchObservedRunningTime="2026-09-17 15:41:20.266383568 +0000 UTC m=+133.228683114"
Sep 17 15:41:20.286711 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:20.286665    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-stkm5" podStartSLOduration=130.751893973 podStartE2EDuration="2m12.286651316s" totalImagesPullingTime="1.534757343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:20.285554081 +0000 UTC m=+133.247853649" watchObservedRunningTime="2026-09-17 15:41:20.286651316 +0000 UTC m=+133.248950862"
Sep 17 15:41:22.138325 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:22.138292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:26.242148 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:26.242116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c6c68f5c-rppcb"
Sep 17 15:41:37.219456 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.219384    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" podUID="fa0a7c8e-f752-455c-95c0-94426646eb76" containerName="registry" containerID="cri-o://575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97" gracePeriod=30
Sep 17 15:41:37.460873 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.460844    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:41:37.596052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.595984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djj8z\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-kube-api-access-djj8z\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-installation-pull-secrets\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596260 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-image-registry-private-configuration\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596260 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596129    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-bound-sa-token\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596260 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-certificates\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596260 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa0a7c8e-f752-455c-95c0-94426646eb76-ca-trust-extracted\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596260 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-trusted-ca\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596517 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") pod \"fa0a7c8e-f752-455c-95c0-94426646eb76\" (UID: \"fa0a7c8e-f752-455c-95c0-94426646eb76\") "
Sep 17 15:41:37.596647 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:37.596865 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.596652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:37.599063 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.599034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-kube-api-access-djj8z" (OuterVolumeSpecName: "kube-api-access-djj8z") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "kube-api-access-djj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:37.599183 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.599034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:37.599183 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.599098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:37.599183 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.599100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:37.599183 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.599174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:37.604489 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.604465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a7c8e-f752-455c-95c0-94426646eb76-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fa0a7c8e-f752-455c-95c0-94426646eb76" (UID: "fa0a7c8e-f752-455c-95c0-94426646eb76"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:41:37.697372 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697353    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djj8z\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-kube-api-access-djj8z\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697372 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697373    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-installation-pull-secrets\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697383    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa0a7c8e-f752-455c-95c0-94426646eb76-image-registry-private-configuration\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697392    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-bound-sa-token\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697401    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-certificates\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697409    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa0a7c8e-f752-455c-95c0-94426646eb76-ca-trust-extracted\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697417    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa0a7c8e-f752-455c-95c0-94426646eb76-trusted-ca\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:37.697486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:37.697425    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa0a7c8e-f752-455c-95c0-94426646eb76-registry-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:38.312920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.312885    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa0a7c8e-f752-455c-95c0-94426646eb76" containerID="575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97" exitCode=0
Sep 17 15:41:38.313298 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.312939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh"
Sep 17 15:41:38.313298 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.312960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" event={"ID":"fa0a7c8e-f752-455c-95c0-94426646eb76","Type":"ContainerDied","Data":"575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97"}
Sep 17 15:41:38.313298 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.313014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcff994bc-brfvh" event={"ID":"fa0a7c8e-f752-455c-95c0-94426646eb76","Type":"ContainerDied","Data":"4ba90a45e079a1903c5084eafd76365c5527aa0b7b3144b9834e060908d7bc50"}
Sep 17 15:41:38.313821 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.313800    2563 scope.go:117] "RemoveContainer" containerID="575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97"
Sep 17 15:41:38.328870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.328848    2563 scope.go:117] "RemoveContainer" containerID="575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97"
Sep 17 15:41:38.329199 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:41:38.329175    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97\": container with ID starting with 575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97 not found: ID does not exist" containerID="575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97"
Sep 17 15:41:38.329300 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.329208    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97"} err="failed to get container status \"575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97\": rpc error: code = NotFound desc = could not find container \"575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97\": container with ID starting with 575ec4a43e77ed105b56d606e87a254f0ba2d796af8a1d453aa1209dcbf4ad97 not found: ID does not exist"
Sep 17 15:41:38.342030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.342004    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dcff994bc-brfvh"]
Sep 17 15:41:38.348275 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:38.348232    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dcff994bc-brfvh"]
Sep 17 15:41:39.534436 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:39.534401    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0a7c8e-f752-455c-95c0-94426646eb76" path="/var/lib/kubelet/pods/fa0a7c8e-f752-455c-95c0-94426646eb76/volumes"
Sep 17 15:41:44.242552 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.242498    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78cf84dc48-nrs9r" podUID="31867d5c-95a6-4e40-b056-d90b98617819" containerName="console" containerID="cri-o://89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247" gracePeriod=15
Sep 17 15:41:44.509447 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.509427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:44.646926 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.646895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-service-ca\") pod \"31867d5c-95a6-4e40-b056-d90b98617819\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") "
Sep 17 15:41:44.647087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.646935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-console-config\") pod \"31867d5c-95a6-4e40-b056-d90b98617819\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") "
Sep 17 15:41:44.647087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.646974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-oauth-config\") pod \"31867d5c-95a6-4e40-b056-d90b98617819\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") "
Sep 17 15:41:44.647087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.646995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjpxw\" (UniqueName: \"kubernetes.io/projected/31867d5c-95a6-4e40-b056-d90b98617819-kube-api-access-jjpxw\") pod \"31867d5c-95a6-4e40-b056-d90b98617819\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") "
Sep 17 15:41:44.647087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.647012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-oauth-serving-cert\") pod \"31867d5c-95a6-4e40-b056-d90b98617819\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") "
Sep 17 15:41:44.647087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.647065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-serving-cert\") pod \"31867d5c-95a6-4e40-b056-d90b98617819\" (UID: \"31867d5c-95a6-4e40-b056-d90b98617819\") "
Sep 17 15:41:44.647392 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.647368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-service-ca" (OuterVolumeSpecName: "service-ca") pod "31867d5c-95a6-4e40-b056-d90b98617819" (UID: "31867d5c-95a6-4e40-b056-d90b98617819"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:44.647524 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.647496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "31867d5c-95a6-4e40-b056-d90b98617819" (UID: "31867d5c-95a6-4e40-b056-d90b98617819"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:44.647581 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.647507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-console-config" (OuterVolumeSpecName: "console-config") pod "31867d5c-95a6-4e40-b056-d90b98617819" (UID: "31867d5c-95a6-4e40-b056-d90b98617819"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:44.649294 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.649272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "31867d5c-95a6-4e40-b056-d90b98617819" (UID: "31867d5c-95a6-4e40-b056-d90b98617819"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:44.649393 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.649371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "31867d5c-95a6-4e40-b056-d90b98617819" (UID: "31867d5c-95a6-4e40-b056-d90b98617819"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:44.649433 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.649388    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31867d5c-95a6-4e40-b056-d90b98617819-kube-api-access-jjpxw" (OuterVolumeSpecName: "kube-api-access-jjpxw") pod "31867d5c-95a6-4e40-b056-d90b98617819" (UID: "31867d5c-95a6-4e40-b056-d90b98617819"). InnerVolumeSpecName "kube-api-access-jjpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:44.747596 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.747571    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-console-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:44.747596 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.747595    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-oauth-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:44.747767 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.747609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjpxw\" (UniqueName: \"kubernetes.io/projected/31867d5c-95a6-4e40-b056-d90b98617819-kube-api-access-jjpxw\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:44.747767 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.747622    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-oauth-serving-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:44.747767 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.747634    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31867d5c-95a6-4e40-b056-d90b98617819-console-serving-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:44.747767 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:44.747647    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31867d5c-95a6-4e40-b056-d90b98617819-service-ca\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:41:45.344638 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.344599    2563 generic.go:356] "Generic (PLEG): container finished" podID="31867d5c-95a6-4e40-b056-d90b98617819" containerID="89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247" exitCode=0
Sep 17 15:41:45.345106 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.344680    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cf84dc48-nrs9r"
Sep 17 15:41:45.345106 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.344697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cf84dc48-nrs9r" event={"ID":"31867d5c-95a6-4e40-b056-d90b98617819","Type":"ContainerDied","Data":"89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247"}
Sep 17 15:41:45.345106 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.344791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cf84dc48-nrs9r" event={"ID":"31867d5c-95a6-4e40-b056-d90b98617819","Type":"ContainerDied","Data":"d76e61cc68d5c65a98f02bdd6c7f52125ef82a3bb8e147e210cd531042137fd8"}
Sep 17 15:41:45.345106 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.344815    2563 scope.go:117] "RemoveContainer" containerID="89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247"
Sep 17 15:41:45.355001 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.354975    2563 scope.go:117] "RemoveContainer" containerID="89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247"
Sep 17 15:41:45.355350 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:41:45.355322    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247\": container with ID starting with 89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247 not found: ID does not exist" containerID="89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247"
Sep 17 15:41:45.355431 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.355361    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247"} err="failed to get container status \"89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247\": rpc error: code = NotFound desc = could not find container \"89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247\": container with ID starting with 89e58793df54405b7f7cb50be29e4dc4e7a46eaa73e930bb08b4753ed6b4c247 not found: ID does not exist"
Sep 17 15:41:45.367825 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.367799    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78cf84dc48-nrs9r"]
Sep 17 15:41:45.374696 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.374671    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78cf84dc48-nrs9r"]
Sep 17 15:41:45.534136 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:41:45.534109    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31867d5c-95a6-4e40-b056-d90b98617819" path="/var/lib/kubelet/pods/31867d5c-95a6-4e40-b056-d90b98617819/volumes"
Sep 17 15:42:04.461523 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:04.461422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:04.478624 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:04.478589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:05.461484 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:05.461452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:18.394836 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.394804    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:18.395296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.395096    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="alertmanager" containerID="cri-o://171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3" gracePeriod=120
Sep 17 15:42:18.395296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.395128    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-web" containerID="cri-o://2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254" gracePeriod=120
Sep 17 15:42:18.395296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.395099    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy" containerID="cri-o://59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559" gracePeriod=120
Sep 17 15:42:18.395296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.395143    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="prom-label-proxy" containerID="cri-o://2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58" gracePeriod=120
Sep 17 15:42:18.395296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.395174    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="config-reloader" containerID="cri-o://eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa" gracePeriod=120
Sep 17 15:42:18.395296 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:18.395174    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-metric" containerID="cri-o://337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c" gracePeriod=120
Sep 17 15:42:19.510204 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510172    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58" exitCode=0
Sep 17 15:42:19.510204 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510201    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c" exitCode=0
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510212    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559" exitCode=0
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510219    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa" exitCode=0
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510224    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3" exitCode=0
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58"}
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c"}
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559"}
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa"}
Sep 17 15:42:19.510572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.510505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3"}
Sep 17 15:42:19.637717 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.637696    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:19.725787 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-web\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.725937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.725937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.725937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-web-config\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.725937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-cluster-tls-config\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.725937 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725953    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-trusted-ca-bundle\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.725990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-main-db\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-metrics-client-ca\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-config-out\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-config-volume\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726150    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-tls-assets\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726215 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrx8z\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-kube-api-access-rrx8z\") pod \"f377e2ee-3090-4835-b030-74c98b295644\" (UID: \"f377e2ee-3090-4835-b030-74c98b295644\") "
Sep 17 15:42:19.726646 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:19.726646 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.726462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:19.728875 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.728793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.729147 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.729028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:19.729147 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.729090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.729486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.729463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-kube-api-access-rrx8z" (OuterVolumeSpecName: "kube-api-access-rrx8z") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "kube-api-access-rrx8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:19.729724 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.729685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.729724 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.729715    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-config-out" (OuterVolumeSpecName: "config-out") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:19.730736 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.730684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-config-volume" (OuterVolumeSpecName: "config-volume") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.730980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.730957    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:19.731888 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.731851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.734617 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.734589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.740998 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.740974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-web-config" (OuterVolumeSpecName: "web-config") pod "f377e2ee-3090-4835-b030-74c98b295644" (UID: "f377e2ee-3090-4835-b030-74c98b295644"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:19.827176 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827153    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-config-volume\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827176 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827174    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-tls-assets\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827183    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrx8z\" (UniqueName: \"kubernetes.io/projected/f377e2ee-3090-4835-b030-74c98b295644-kube-api-access-rrx8z\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827194    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827203    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827212    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827222    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-web-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827230    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-cluster-tls-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827259    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f377e2ee-3090-4835-b030-74c98b295644-secret-alertmanager-main-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827268    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827276    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-alertmanager-main-db\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827285    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f377e2ee-3090-4835-b030-74c98b295644-metrics-client-ca\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:19.827319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:19.827292    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f377e2ee-3090-4835-b030-74c98b295644-config-out\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:20.517650 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.517622    2563 generic.go:356] "Generic (PLEG): container finished" podID="f377e2ee-3090-4835-b030-74c98b295644" containerID="2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254" exitCode=0
Sep 17 15:42:20.517938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.517709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254"}
Sep 17 15:42:20.517938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.517735    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.517938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.517836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f377e2ee-3090-4835-b030-74c98b295644","Type":"ContainerDied","Data":"b310809293b683a7afed715e1763f8d5e3af8c0960526d8fbbf61f01e22ce862"}
Sep 17 15:42:20.517938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.517890    2563 scope.go:117] "RemoveContainer" containerID="2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58"
Sep 17 15:42:20.526505 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.526489    2563 scope.go:117] "RemoveContainer" containerID="337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c"
Sep 17 15:42:20.533372 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.533351    2563 scope.go:117] "RemoveContainer" containerID="59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559"
Sep 17 15:42:20.539903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.539886    2563 scope.go:117] "RemoveContainer" containerID="2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254"
Sep 17 15:42:20.546623 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.546606    2563 scope.go:117] "RemoveContainer" containerID="eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa"
Sep 17 15:42:20.549307 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.549285    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:20.555268 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.555228    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:20.555652 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.555632    2563 scope.go:117] "RemoveContainer" containerID="171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3"
Sep 17 15:42:20.562391 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.562376    2563 scope.go:117] "RemoveContainer" containerID="b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272"
Sep 17 15:42:20.568938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.568924    2563 scope.go:117] "RemoveContainer" containerID="2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58"
Sep 17 15:42:20.569190 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.569173    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58\": container with ID starting with 2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58 not found: ID does not exist" containerID="2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58"
Sep 17 15:42:20.569259 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.569197    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58"} err="failed to get container status \"2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58\": rpc error: code = NotFound desc = could not find container \"2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58\": container with ID starting with 2f45869a44bf2c4cca393865e42c01142e57a3757688ef84e83fda47d5466e58 not found: ID does not exist"
Sep 17 15:42:20.569259 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.569212    2563 scope.go:117] "RemoveContainer" containerID="337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c"
Sep 17 15:42:20.569486 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.569472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c\": container with ID starting with 337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c not found: ID does not exist" containerID="337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c"
Sep 17 15:42:20.569524 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.569488    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c"} err="failed to get container status \"337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c\": rpc error: code = NotFound desc = could not find container \"337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c\": container with ID starting with 337334a816ae3183b033a244e4d789f40905af88088d2637b35d44441885d47c not found: ID does not exist"
Sep 17 15:42:20.569524 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.569500    2563 scope.go:117] "RemoveContainer" containerID="59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559"
Sep 17 15:42:20.569729 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.569709    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559\": container with ID starting with 59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559 not found: ID does not exist" containerID="59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559"
Sep 17 15:42:20.569792 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.569737    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559"} err="failed to get container status \"59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559\": rpc error: code = NotFound desc = could not find container \"59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559\": container with ID starting with 59e9ced103953eacc79630670e1c50d2a5687fc957d5040e69cf9ca228f6b559 not found: ID does not exist"
Sep 17 15:42:20.569792 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.569763    2563 scope.go:117] "RemoveContainer" containerID="2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254"
Sep 17 15:42:20.570012 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.569997    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254\": container with ID starting with 2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254 not found: ID does not exist" containerID="2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254"
Sep 17 15:42:20.570073 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254"} err="failed to get container status \"2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254\": rpc error: code = NotFound desc = could not find container \"2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254\": container with ID starting with 2292cc365288a775e0bf1864a2cb05df9df6bb800c4046052988b7dc92eb5254 not found: ID does not exist"
Sep 17 15:42:20.570073 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570035    2563 scope.go:117] "RemoveContainer" containerID="eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa"
Sep 17 15:42:20.570298 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.570279    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa\": container with ID starting with eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa not found: ID does not exist" containerID="eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa"
Sep 17 15:42:20.570353 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570304    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa"} err="failed to get container status \"eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa\": rpc error: code = NotFound desc = could not find container \"eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa\": container with ID starting with eceae0f51200af2bda7f57ce942468b48eb34e44db0d58ce9e939eef3598cffa not found: ID does not exist"
Sep 17 15:42:20.570353 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570318    2563 scope.go:117] "RemoveContainer" containerID="171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3"
Sep 17 15:42:20.570560 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.570543    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3\": container with ID starting with 171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3 not found: ID does not exist" containerID="171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3"
Sep 17 15:42:20.570593 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570566    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3"} err="failed to get container status \"171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3\": rpc error: code = NotFound desc = could not find container \"171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3\": container with ID starting with 171f1c88c9ba88e5a9a324f3368508efeb57d7659b5f511159cd570d937ff6b3 not found: ID does not exist"
Sep 17 15:42:20.570593 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570580    2563 scope.go:117] "RemoveContainer" containerID="b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272"
Sep 17 15:42:20.570800 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:20.570784    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272\": container with ID starting with b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272 not found: ID does not exist" containerID="b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272"
Sep 17 15:42:20.570832 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.570805    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272"} err="failed to get container status \"b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272\": rpc error: code = NotFound desc = could not find container \"b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272\": container with ID starting with b3562422a612c4a4abdc9711f0fc6a283dc591d2ff82d0bbde3388a82eb87272 not found: ID does not exist"
Sep 17 15:42:20.583008 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.582986    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:20.583369 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583352    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31867d5c-95a6-4e40-b056-d90b98617819" containerName="console"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31867d5c-95a6-4e40-b056-d90b98617819" containerName="console"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583383    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583391    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583404    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-web"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583412    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-web"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583425    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="init-config-reloader"
Sep 17 15:42:20.583441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583434    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="init-config-reloader"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583453    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0a7c8e-f752-455c-95c0-94426646eb76" containerName="registry"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0a7c8e-f752-455c-95c0-94426646eb76" containerName="registry"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="prom-label-proxy"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="prom-label-proxy"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583502    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="alertmanager"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="alertmanager"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="config-reloader"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583526    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="config-reloader"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583543    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583551    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583626    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="alertmanager"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583641    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="prom-label-proxy"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31867d5c-95a6-4e40-b056-d90b98617819" containerName="console"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583662    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583671    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0a7c8e-f752-455c-95c0-94426646eb76" containerName="registry"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583680    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="config-reloader"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-web"
Sep 17 15:42:20.583781 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.583697    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f377e2ee-3090-4835-b030-74c98b295644" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:20.591763 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.591742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.594398 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594788    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zzm7n\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:42:20.594908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.594813    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:42:20.600303 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.600283    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:42:20.604630 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.604584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:20.734426 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734426 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734426 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-config-out\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwvx\" (UniqueName: \"kubernetes.io/projected/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-kube-api-access-rhwvx\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-web-config\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734864 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-config-volume\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.734864 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.734736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.835936 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.835909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836031 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.835943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836031 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.835966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836031 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.835986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836031 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836207 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836207 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836207 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-config-out\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836403 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836403 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836403 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwvx\" (UniqueName: \"kubernetes.io/projected/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-kube-api-access-rhwvx\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836403 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-web-config\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.836403 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.836384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-config-volume\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.837216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.837185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.837340 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.837294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.838930 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.838871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.839286 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.839262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-config-volume\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.839395 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.839266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.839667 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.839623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.839667 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.839648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.839779 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.839726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.840034 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.839990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-config-out\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.840155 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.840127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.840338 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.840319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.841356 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.841336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-web-config\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.849432 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.849413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwvx\" (UniqueName: \"kubernetes.io/projected/766f2c5c-47b7-44d1-8dee-45a3a3adaf41-kube-api-access-rhwvx\") pod \"alertmanager-main-0\" (UID: \"766f2c5c-47b7-44d1-8dee-45a3a3adaf41\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:20.900981 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:20.900952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:21.192044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:21.192019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:21.193680 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:42:21.193651    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766f2c5c_47b7_44d1_8dee_45a3a3adaf41.slice/crio-259b9ff44c8e527e3f4d952f10444b4ea91109c65fc6a7ec224a82e46193204f WatchSource:0}: Error finding container 259b9ff44c8e527e3f4d952f10444b4ea91109c65fc6a7ec224a82e46193204f: Status 404 returned error can't find the container with id 259b9ff44c8e527e3f4d952f10444b4ea91109c65fc6a7ec224a82e46193204f
Sep 17 15:42:21.523905 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:21.523847    2563 generic.go:356] "Generic (PLEG): container finished" podID="766f2c5c-47b7-44d1-8dee-45a3a3adaf41" containerID="6df0f5ecbf917288f3d9409ea84c7ebb864414c07612634a69909192cfa54300" exitCode=0
Sep 17 15:42:21.524211 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:21.523933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerDied","Data":"6df0f5ecbf917288f3d9409ea84c7ebb864414c07612634a69909192cfa54300"}
Sep 17 15:42:21.524211 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:21.524010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"259b9ff44c8e527e3f4d952f10444b4ea91109c65fc6a7ec224a82e46193204f"}
Sep 17 15:42:21.535321 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:21.535296    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f377e2ee-3090-4835-b030-74c98b295644" path="/var/lib/kubelet/pods/f377e2ee-3090-4835-b030-74c98b295644/volumes"
Sep 17 15:42:22.533694 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.533665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"8e83f6860adeaa60bc1b1919d7a3aeb5ce137b31e7e5e6fd85751ea497a1a4e3"}
Sep 17 15:42:22.534044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.533762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"1feafaf86a1450e6fce388d382349171c7a6b88677c1f9a07243fcddcbe9fce5"}
Sep 17 15:42:22.534044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.533792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"6bb144573fcde9a3e7c05c42f0dc4893ffaade6ce0a9f91299fc6d5b48c098bd"}
Sep 17 15:42:22.534044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.533823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"c27686826f64920e57a72c0d02270fd63dcf053e2b9130f884c2f21a8c16e3b2"}
Sep 17 15:42:22.534044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.533848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"c5e317ca567ce11544fcc360609c0b82a497e7e1fbe3449578c57da6acd1208c"}
Sep 17 15:42:22.534044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.533871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"766f2c5c-47b7-44d1-8dee-45a3a3adaf41","Type":"ContainerStarted","Data":"7570699c59586316a261749f7c4a8c0587b1fd6fa38d4803954fd3a52a2161cd"}
Sep 17 15:42:22.566908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.566861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.551379886 podStartE2EDuration="2.566844816s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.46493ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:22.564316923 +0000 UTC m=+195.526616497" watchObservedRunningTime="2026-09-17 15:42:22.566844816 +0000 UTC m=+195.529144736"
Sep 17 15:42:22.668980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.668944    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:22.669380 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.669346    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="prometheus" containerID="cri-o://09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74" gracePeriod=600
Sep 17 15:42:22.669506 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.669375    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="config-reloader" containerID="cri-o://e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b" gracePeriod=600
Sep 17 15:42:22.669506 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.669366    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy" containerID="cri-o://c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc" gracePeriod=600
Sep 17 15:42:22.669506 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.669398    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-web" containerID="cri-o://8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391" gracePeriod=600
Sep 17 15:42:22.669506 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.669500    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f" gracePeriod=600
Sep 17 15:42:22.669727 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.669584    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="thanos-sidecar" containerID="cri-o://d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e" gracePeriod=600
Sep 17 15:42:22.915719 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:22.915689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.055870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.055795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-kube-rbac-proxy\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.055870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.055831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-grpc-tls\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.055870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.055863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-thanos-prometheus-http-client-file\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056121 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.055889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-db\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056121 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056008    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056121 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config-out\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056121 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-tls-assets\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056362 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-kubelet-serving-ca-bundle\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056362 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056362 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056362 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w99pd\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-kube-api-access-w99pd\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056362 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-tls\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056362 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-serving-certs-ca-bundle\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-web-config\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-0\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-trusted-ca-bundle\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056474    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-metrics-client-ca\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-2\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-1\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-metrics-client-certs\") pod \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\" (UID: \"b28846b8-0e84-44ae-bd11-7e8ba96b279b\") "
Sep 17 15:42:23.056940 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.056868    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:23.058105 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.057848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.058264 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.058223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.058722 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.058493    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.058844 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.058745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.058844 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.058783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.059417 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.059107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.059684 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.059605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.059684 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.059649    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.059837 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.059691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config-out" (OuterVolumeSpecName: "config-out") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:23.059837 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.059791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:23.060040 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.060008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.060293 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.060234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.060379 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.060313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.060505 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.060485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.060829 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.060799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-kube-api-access-w99pd" (OuterVolumeSpecName: "kube-api-access-w99pd") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "kube-api-access-w99pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:23.061185 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.061161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.061300 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.061168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:23.061675 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.061657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config" (OuterVolumeSpecName: "config") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.070197 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.070179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-web-config" (OuterVolumeSpecName: "web-config") pod "b28846b8-0e84-44ae-bd11-7e8ba96b279b" (UID: "b28846b8-0e84-44ae-bd11-7e8ba96b279b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:23.157688 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157664    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157688 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157687    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-db\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157697    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157706    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config-out\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157714    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-tls-assets\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157722    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157731    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157738    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157746    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w99pd\" (UniqueName: \"kubernetes.io/projected/b28846b8-0e84-44ae-bd11-7e8ba96b279b-kube-api-access-w99pd\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157755    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157762    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157770    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-web-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157778    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157786    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157794    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-configmap-metrics-client-ca\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157802    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157810    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b28846b8-0e84-44ae-bd11-7e8ba96b279b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.157809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157818    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-metrics-client-certs\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.158286 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157828    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-kube-rbac-proxy\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.158286 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.157836    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b28846b8-0e84-44ae-bd11-7e8ba96b279b-secret-grpc-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:42:23.542196 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542166    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f" exitCode=0
Sep 17 15:42:23.542196 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542193    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc" exitCode=0
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542203    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391" exitCode=0
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542211    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e" exitCode=0
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542218    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b" exitCode=0
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542227    2563 generic.go:356] "Generic (PLEG): container finished" podID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74" exitCode=0
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542266    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"}
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"}
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"}
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"}
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542487    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.542644 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"}
Sep 17 15:42:23.543284 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"}
Sep 17 15:42:23.543284 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.542750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b28846b8-0e84-44ae-bd11-7e8ba96b279b","Type":"ContainerDied","Data":"97c81d44cf4f828524c02390d5068d0c729c682ff3df8f26eb0947d9f85d3198"}
Sep 17 15:42:23.550179 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.550163    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.556857 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.556840    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.563728 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.563699    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.569279 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.569257    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:23.570964 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.570950    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.575083 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.575064    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:23.577753 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.577739    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.589419 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.589376    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.596751 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.596731    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.597020 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.597000    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.597104 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597031    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"} err="failed to get container status \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist"
Sep 17 15:42:23.597104 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597056    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.597337 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.597319    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.597389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597342    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"} err="failed to get container status \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist"
Sep 17 15:42:23.597389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597357    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.597553 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.597538    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.597589 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597554    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"} err="failed to get container status \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist"
Sep 17 15:42:23.597589 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597566    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.597801 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.597785    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.597862 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"} err="failed to get container status \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist"
Sep 17 15:42:23.597862 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.597818    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.598067 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.598051    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.598105 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598072    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"} err="failed to get container status \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist"
Sep 17 15:42:23.598105 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598087    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.598349 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.598333    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.598396 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598352    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"} err="failed to get container status \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist"
Sep 17 15:42:23.598396 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598365    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.598621 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:42:23.598607    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.598675 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598621    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"} err="failed to get container status \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist"
Sep 17 15:42:23.598675 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598632    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.598835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598813    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"} err="failed to get container status \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist"
Sep 17 15:42:23.598876 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.598835    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.599036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599020    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"} err="failed to get container status \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist"
Sep 17 15:42:23.599036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599035    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.599223 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599208    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"} err="failed to get container status \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist"
Sep 17 15:42:23.599223 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599222    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.599474 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599455    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"} err="failed to get container status \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist"
Sep 17 15:42:23.599551 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599475    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.599686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599669    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"} err="failed to get container status \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist"
Sep 17 15:42:23.599726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599686    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.599880 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599863    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"} err="failed to get container status \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist"
Sep 17 15:42:23.599945 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.599882    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.600125 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600109    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"} err="failed to get container status \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist"
Sep 17 15:42:23.600180 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600125    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.600400 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600382    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"} err="failed to get container status \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist"
Sep 17 15:42:23.600469 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600401    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.600603 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600586    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"} err="failed to get container status \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist"
Sep 17 15:42:23.600645 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600603    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.600815 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600798    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"} err="failed to get container status \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist"
Sep 17 15:42:23.600886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.600817    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.601046 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601029    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"} err="failed to get container status \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist"
Sep 17 15:42:23.601093 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601046    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.601288 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601234    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"} err="failed to get container status \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist"
Sep 17 15:42:23.601363 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601290    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.601531 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601513    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"} err="failed to get container status \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist"
Sep 17 15:42:23.601579 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601532    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.601737 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601719    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"} err="failed to get container status \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist"
Sep 17 15:42:23.601776 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601737    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.601942 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601925    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"} err="failed to get container status \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist"
Sep 17 15:42:23.602002 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.601944    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.602156 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602139    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"} err="failed to get container status \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist"
Sep 17 15:42:23.602202 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602156    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.602377 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602360    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"} err="failed to get container status \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist"
Sep 17 15:42:23.602427 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602378    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.602584 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602566    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"} err="failed to get container status \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist"
Sep 17 15:42:23.602629 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602585    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.602799 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602783    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"} err="failed to get container status \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist"
Sep 17 15:42:23.602838 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602799    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.603000 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.602985    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"} err="failed to get container status \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist"
Sep 17 15:42:23.603038 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603000    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.603230 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603207    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"} err="failed to get container status \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist"
Sep 17 15:42:23.603319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603230    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.603504 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603485    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"} err="failed to get container status \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist"
Sep 17 15:42:23.603556 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603505    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.603734 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603714    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"} err="failed to get container status \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist"
Sep 17 15:42:23.603797 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603735    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.603986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603967    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"} err="failed to get container status \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist"
Sep 17 15:42:23.604036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.603987    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.604217 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604197    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"} err="failed to get container status \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist"
Sep 17 15:42:23.604306 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604219    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.604445 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604429    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"} err="failed to get container status \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist"
Sep 17 15:42:23.604492 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604447    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.604648 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604628    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"} err="failed to get container status \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist"
Sep 17 15:42:23.604719 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604650    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.604856 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"} err="failed to get container status \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist"
Sep 17 15:42:23.604902 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.604857    2563 scope.go:117] "RemoveContainer" containerID="741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"
Sep 17 15:42:23.605284 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.605220    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f"} err="failed to get container status \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": rpc error: code = NotFound desc = could not find container \"741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f\": container with ID starting with 741b184667ef656d3a55b9f276147c35f95b720f6aaf1138abb968ac28b34f6f not found: ID does not exist"
Sep 17 15:42:23.605284 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.605271    2563 scope.go:117] "RemoveContainer" containerID="c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"
Sep 17 15:42:23.605924 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.605862    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc"} err="failed to get container status \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": rpc error: code = NotFound desc = could not find container \"c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc\": container with ID starting with c8ff474379d51221660629f1e948428a6bee44fca581d00c6b6201b4105f33fc not found: ID does not exist"
Sep 17 15:42:23.605924 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.605889    2563 scope.go:117] "RemoveContainer" containerID="8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"
Sep 17 15:42:23.606167 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606144    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391"} err="failed to get container status \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": rpc error: code = NotFound desc = could not find container \"8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391\": container with ID starting with 8dc716635bd06a5714237208f5f09000665f496430d233e92d85747ee01bd391 not found: ID does not exist"
Sep 17 15:42:23.606217 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606176    2563 scope.go:117] "RemoveContainer" containerID="d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"
Sep 17 15:42:23.606438 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606411    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e"} err="failed to get container status \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": rpc error: code = NotFound desc = could not find container \"d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e\": container with ID starting with d182bc28162774abbb80eab62d5b7485eb9bbcbc688c6eb56316f9a17567355e not found: ID does not exist"
Sep 17 15:42:23.606438 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606437    2563 scope.go:117] "RemoveContainer" containerID="e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"
Sep 17 15:42:23.606696 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606678    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b"} err="failed to get container status \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": rpc error: code = NotFound desc = could not find container \"e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b\": container with ID starting with e7b84256c4649f7da70e477d78cd43f8d4f07a9ff7da65105af5eff8f121c94b not found: ID does not exist"
Sep 17 15:42:23.606750 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606705    2563 scope.go:117] "RemoveContainer" containerID="09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"
Sep 17 15:42:23.606948 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606931    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74"} err="failed to get container status \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": rpc error: code = NotFound desc = could not find container \"09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74\": container with ID starting with 09bebe82f9c4151bfd7e90f7e29de6f7eaac2a03b5bbd2b4b316a1e5a2871e74 not found: ID does not exist"
Sep 17 15:42:23.606992 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.606948    2563 scope.go:117] "RemoveContainer" containerID="9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"
Sep 17 15:42:23.607155 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607134    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:23.607234 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607159    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b"} err="failed to get container status \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": rpc error: code = NotFound desc = could not find container \"9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b\": container with ID starting with 9bc18eb82c21e8483fd639d37bf0a5c127d138cd008a340369e1eb97de39ab2b not found: ID does not exist"
Sep 17 15:42:23.607493 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="thanos-sidecar"
Sep 17 15:42:23.607528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607494    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="thanos-sidecar"
Sep 17 15:42:23.607528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607503    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-web"
Sep 17 15:42:23.607528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-web"
Sep 17 15:42:23.607528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="init-config-reloader"
Sep 17 15:42:23.607528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607523    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="init-config-reloader"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607530    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607545    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="config-reloader"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607550    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="config-reloader"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607556    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-thanos"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-thanos"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607569    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="prometheus"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607573    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="prometheus"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="thanos-sidecar"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607634    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-thanos"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="prometheus"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607648    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="config-reloader"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607653    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy"
Sep 17 15:42:23.607671 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.607659    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" containerName="kube-rbac-proxy-web"
Sep 17 15:42:23.611905 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.611887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.614995 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.614975    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:42:23.615093 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:42:23.615139 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:42:23.615188 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:42:23.615275 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615232    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:42:23.615413 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:42:23.615467 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6znx\""
Sep 17 15:42:23.615559 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:42:23.615685 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:42:23.615685 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82bolsmjilt5v\""
Sep 17 15:42:23.615828 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615740    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:42:23.615952 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:42:23.616047 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:42:23.616047 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.615984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:42:23.616782 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.616768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:42:23.620326 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.620303    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:42:23.622122 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.622106    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:42:23.630543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.630522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:23.762157 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762157 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762319 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762472 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0917d62-1786-4f5c-8fff-42dbf2769408-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762472 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762472 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762472 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-config\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762472 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0917d62-1786-4f5c-8fff-42dbf2769408-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxtb\" (UniqueName: \"kubernetes.io/projected/f0917d62-1786-4f5c-8fff-42dbf2769408-kube-api-access-msxtb\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762686 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.762909 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.762703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863590 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863743 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863743 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.863743 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.863733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0917d62-1786-4f5c-8fff-42dbf2769408-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864193 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864193 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-config\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864193 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864193 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864193 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0917d62-1786-4f5c-8fff-42dbf2769408-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxtb\" (UniqueName: \"kubernetes.io/projected/f0917d62-1786-4f5c-8fff-42dbf2769408-kube-api-access-msxtb\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864478 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.864965 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.864529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.865059 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.865033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.865200 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.865125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.865793 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.865761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.866726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.866701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.866960 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.866936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.867270 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.867175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.867515 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.867488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.867597 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.867515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-config\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.868269 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.867915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0917d62-1786-4f5c-8fff-42dbf2769408-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.868269 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.868174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0917d62-1786-4f5c-8fff-42dbf2769408-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.868412 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.868292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.868820 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.868791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0917d62-1786-4f5c-8fff-42dbf2769408-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.869773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.869750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.869842 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.869788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.869842 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.869824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.870100 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.870080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0917d62-1786-4f5c-8fff-42dbf2769408-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.877445 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.877424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxtb\" (UniqueName: \"kubernetes.io/projected/f0917d62-1786-4f5c-8fff-42dbf2769408-kube-api-access-msxtb\") pod \"prometheus-k8s-0\" (UID: \"f0917d62-1786-4f5c-8fff-42dbf2769408\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:23.922247 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:23.922226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:24.204826 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:24.204800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:24.206419 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:42:24.206394    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0917d62_1786_4f5c_8fff_42dbf2769408.slice/crio-e6eb2e938a2fb539b26d1ddb3f06a1503d11f4559b8e4cd9259047efbc5631f3 WatchSource:0}: Error finding container e6eb2e938a2fb539b26d1ddb3f06a1503d11f4559b8e4cd9259047efbc5631f3: Status 404 returned error can't find the container with id e6eb2e938a2fb539b26d1ddb3f06a1503d11f4559b8e4cd9259047efbc5631f3
Sep 17 15:42:24.550324 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:24.550259    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0917d62-1786-4f5c-8fff-42dbf2769408" containerID="7b7702fd4c8247945dc1295c2da2d0a44a8fe39fee2e407dd314563408932d31" exitCode=0
Sep 17 15:42:24.550651 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:24.550350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerDied","Data":"7b7702fd4c8247945dc1295c2da2d0a44a8fe39fee2e407dd314563408932d31"}
Sep 17 15:42:24.550651 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:24.550444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"e6eb2e938a2fb539b26d1ddb3f06a1503d11f4559b8e4cd9259047efbc5631f3"}
Sep 17 15:42:25.534762 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.534726    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28846b8-0e84-44ae-bd11-7e8ba96b279b" path="/var/lib/kubelet/pods/b28846b8-0e84-44ae-bd11-7e8ba96b279b/volumes"
Sep 17 15:42:25.560638 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.560606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"996c1ec79066ac0561d0034adf286725ed73fba790fda7b4ab50a7a67e7ff492"}
Sep 17 15:42:25.560945 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.560672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"dab371c961cb83f373bd44f2e49691d64b3520b05b72818b862b3f6c7dbba96b"}
Sep 17 15:42:25.560945 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.560700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"40831b04d0ebfc683d41f5f90d9e21d2e37409f24f499ff4feeca918bebc3a33"}
Sep 17 15:42:25.560945 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.560730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"60daec1f8b93b818b6a81463b85ac956f639c94295ad68712a680d7ba662106b"}
Sep 17 15:42:25.560945 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.560753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"2484c72e0bd73f3e1b17536fee44bdb026b9066590e774823a6b553c06847860"}
Sep 17 15:42:25.560945 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.560778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0917d62-1786-4f5c-8fff-42dbf2769408","Type":"ContainerStarted","Data":"31924a6967339a616f43601e29f93cbdcbe81e35cd0f9aec872b9393ac194ecd"}
Sep 17 15:42:25.613507 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:25.613455    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.592079767 podStartE2EDuration="2.61343952s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.359753ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:25.613051363 +0000 UTC m=+198.575350937" watchObservedRunningTime="2026-09-17 15:42:25.61343952 +0000 UTC m=+198.575739125"
Sep 17 15:42:28.923290 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:42:28.923223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:23.922660 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:43:23.922608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:23.938378 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:43:23.938352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:24.835771 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:43:24.835741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:44:07.421756 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:44:07.421726    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:48:28.922470 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.922427    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-lszj8"]
Sep 17 15:48:28.926213 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.926182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lszj8"
Sep 17 15:48:28.928499 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.928473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 15:48:28.928906 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.928887    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6vf9l\""
Sep 17 15:48:28.928979 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.928942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:48:28.929507 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.929490    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 15:48:28.929585 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.929527    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:48:28.934161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.934137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lszj8"]
Sep 17 15:48:28.971638 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.971614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-model-cache\") pod \"s3-init-lszj8\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") " pod="kserve/s3-init-lszj8"
Sep 17 15:48:28.971737 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:28.971653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9pv\" (UniqueName: \"kubernetes.io/projected/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-kube-api-access-rr9pv\") pod \"s3-init-lszj8\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") " pod="kserve/s3-init-lszj8"
Sep 17 15:48:29.072702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.072677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-model-cache\") pod \"s3-init-lszj8\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") " pod="kserve/s3-init-lszj8"
Sep 17 15:48:29.072811 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.072714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9pv\" (UniqueName: \"kubernetes.io/projected/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-kube-api-access-rr9pv\") pod \"s3-init-lszj8\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") " pod="kserve/s3-init-lszj8"
Sep 17 15:48:29.073054 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.073030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-model-cache\") pod \"s3-init-lszj8\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") " pod="kserve/s3-init-lszj8"
Sep 17 15:48:29.081934 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.081914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9pv\" (UniqueName: \"kubernetes.io/projected/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-kube-api-access-rr9pv\") pod \"s3-init-lszj8\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") " pod="kserve/s3-init-lszj8"
Sep 17 15:48:29.244683 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.244627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lszj8"
Sep 17 15:48:29.510958 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.510936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lszj8"]
Sep 17 15:48:29.513071 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:48:29.513026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9ad5245_7b7e_4ced_b3f1_0be9be0e481e.slice/crio-1040b8ba2c5f707f8b621ba517597ccc07f2db8155c3a673c527c2dc25d1aa26 WatchSource:0}: Error finding container 1040b8ba2c5f707f8b621ba517597ccc07f2db8155c3a673c527c2dc25d1aa26: Status 404 returned error can't find the container with id 1040b8ba2c5f707f8b621ba517597ccc07f2db8155c3a673c527c2dc25d1aa26
Sep 17 15:48:29.515026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:29.515007    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:48:30.124360 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:30.124329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lszj8" event={"ID":"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e","Type":"ContainerStarted","Data":"1040b8ba2c5f707f8b621ba517597ccc07f2db8155c3a673c527c2dc25d1aa26"}
Sep 17 15:48:34.151577 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:34.151539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lszj8" event={"ID":"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e","Type":"ContainerStarted","Data":"152b89af55f867064b59a5a92a755dfc93a1ff943a1742587b570cc69389e6dc"}
Sep 17 15:48:36.162060 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:36.162023    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerID="152b89af55f867064b59a5a92a755dfc93a1ff943a1742587b570cc69389e6dc" exitCode=0
Sep 17 15:48:36.162429 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:36.162110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lszj8" event={"ID":"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e","Type":"ContainerDied","Data":"152b89af55f867064b59a5a92a755dfc93a1ff943a1742587b570cc69389e6dc"}
Sep 17 15:48:41.188221 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:41.188171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lszj8" event={"ID":"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e","Type":"ContainerStarted","Data":"c96d7df8de69d645becbbec921d531ea91e5aeec9720dbbc69f169ba0e85d5fc"}
Sep 17 15:48:41.205367 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:48:41.205320    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-lszj8" podStartSLOduration=2.710265588 podStartE2EDuration="13.205307312s" totalImagesPullingTime="8.362756217s" totalInitContainerRuntime="2.132285507s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:41.204121373 +0000 UTC m=+574.166420919" watchObservedRunningTime="2026-09-17 15:48:41.205307312 +0000 UTC m=+574.167606857"
Sep 17 15:49:46.471732 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:46.471698    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerID="c96d7df8de69d645becbbec921d531ea91e5aeec9720dbbc69f169ba0e85d5fc" exitCode=0
Sep 17 15:49:46.472137 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:46.471744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lszj8" event={"ID":"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e","Type":"ContainerDied","Data":"c96d7df8de69d645becbbec921d531ea91e5aeec9720dbbc69f169ba0e85d5fc"}
Sep 17 15:49:47.605847 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.605825    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lszj8"
Sep 17 15:49:47.686640 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.686604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-model-cache\") pod \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") "
Sep 17 15:49:47.686755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.686700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr9pv\" (UniqueName: \"kubernetes.io/projected/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-kube-api-access-rr9pv\") pod \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\" (UID: \"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e\") "
Sep 17 15:49:47.688726 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.688703    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-kube-api-access-rr9pv" (OuterVolumeSpecName: "kube-api-access-rr9pv") pod "c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" (UID: "c9ad5245-7b7e-4ced-b3f1-0be9be0e481e"). InnerVolumeSpecName "kube-api-access-rr9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:47.704819 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.704798    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" (UID: "c9ad5245-7b7e-4ced-b3f1-0be9be0e481e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:47.787564 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.787513    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-model-cache\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:49:47.787564 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:47.787533    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9pv\" (UniqueName: \"kubernetes.io/projected/c9ad5245-7b7e-4ced-b3f1-0be9be0e481e-kube-api-access-rr9pv\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:49:48.481181 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:48.481152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lszj8"
Sep 17 15:49:48.481181 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:48.481159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lszj8" event={"ID":"c9ad5245-7b7e-4ced-b3f1-0be9be0e481e","Type":"ContainerDied","Data":"1040b8ba2c5f707f8b621ba517597ccc07f2db8155c3a673c527c2dc25d1aa26"}
Sep 17 15:49:48.481417 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:49:48.481195    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1040b8ba2c5f707f8b621ba517597ccc07f2db8155c3a673c527c2dc25d1aa26"
Sep 17 15:50:02.060176 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.060144    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"]
Sep 17 15:50:02.060543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.060527    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerName="s3-init"
Sep 17 15:50:02.060543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.060539    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerName="s3-init"
Sep 17 15:50:02.060618 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.060549    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerName="download-hf-model"
Sep 17 15:50:02.060618 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.060555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerName="download-hf-model"
Sep 17 15:50:02.060682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.060620    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ad5245-7b7e-4ced-b3f1-0be9be0e481e" containerName="s3-init"
Sep 17 15:50:02.063972 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.063952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.066891 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.066866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 15:50:02.066891 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.066882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 15:50:02.067062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.066866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:50:02.067062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.066940    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:50:02.067062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.066991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 15:50:02.067179 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.067141    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:50:02.067935 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.067921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:50:02.080171 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.080147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"]
Sep 17 15:50:02.159662 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.159639    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"]
Sep 17 15:50:02.163712 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.163696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.166034 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.166014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 15:50:02.175621 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.175600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"]
Sep 17 15:50:02.191915 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.191889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.191999 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.191956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.191999 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.191976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.192074 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.192052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7de2c36c-d297-42a9-9f86-4a8c09e6e793-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.192114 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.192088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gs7\" (UniqueName: \"kubernetes.io/projected/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-api-access-m9gs7\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.292633 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469qp\" (UniqueName: \"kubernetes.io/projected/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-api-access-469qp\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.292633 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.292903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7de2c36c-d297-42a9-9f86-4a8c09e6e793-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.292903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gs7\" (UniqueName: \"kubernetes.io/projected/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-api-access-m9gs7\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.292903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.292903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.292903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.292903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.293127 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.292979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.293127 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.293008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.293394 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.293373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.293576 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.293557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.293615 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.293580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.295023 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.295005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7de2c36c-d297-42a9-9f86-4a8c09e6e793-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.302857 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.302834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gs7\" (UniqueName: \"kubernetes.io/projected/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-api-access-m9gs7\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.374182 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.374164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:02.394350 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.394320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469qp\" (UniqueName: \"kubernetes.io/projected/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-api-access-469qp\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.394465 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.394361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.394465 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.394413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.394465 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.394438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.394632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.394481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.394977 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.394944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.395086 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.395055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.395200 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.395180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.396753 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.396734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.405636 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.405611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-469qp\" (UniqueName: \"kubernetes.io/projected/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-api-access-469qp\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-v6nkg\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.473251 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.473219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:02.669803 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.669780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"]
Sep 17 15:50:02.671256 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:50:02.671218    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de2c36c_d297_42a9_9f86_4a8c09e6e793.slice/crio-5c8a27c7739a8d5f099a22b5a747d7433a985d6fb117a434b0ff25336b46dbb8 WatchSource:0}: Error finding container 5c8a27c7739a8d5f099a22b5a747d7433a985d6fb117a434b0ff25336b46dbb8: Status 404 returned error can't find the container with id 5c8a27c7739a8d5f099a22b5a747d7433a985d6fb117a434b0ff25336b46dbb8
Sep 17 15:50:02.752306 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:02.752278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"]
Sep 17 15:50:02.753407 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:50:02.753379    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9ff287_f9dd_4b6a_a60b_94e08c6c5c23.slice/crio-3ac0b7e77b4dc766b8a6909f1669aceb4017b14aa2dd489543176f3aa1e1fef6 WatchSource:0}: Error finding container 3ac0b7e77b4dc766b8a6909f1669aceb4017b14aa2dd489543176f3aa1e1fef6: Status 404 returned error can't find the container with id 3ac0b7e77b4dc766b8a6909f1669aceb4017b14aa2dd489543176f3aa1e1fef6
Sep 17 15:50:03.553424 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:03.553389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerStarted","Data":"40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9"}
Sep 17 15:50:03.553814 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:03.553459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerStarted","Data":"3ac0b7e77b4dc766b8a6909f1669aceb4017b14aa2dd489543176f3aa1e1fef6"}
Sep 17 15:50:03.554866 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:03.554842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerStarted","Data":"74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5"}
Sep 17 15:50:03.554916 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:03.554899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerStarted","Data":"5c8a27c7739a8d5f099a22b5a747d7433a985d6fb117a434b0ff25336b46dbb8"}
Sep 17 15:50:06.569431 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:06.569402    2563 generic.go:356] "Generic (PLEG): container finished" podID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerID="74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5" exitCode=0
Sep 17 15:50:06.569840 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:06.569483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerDied","Data":"74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5"}
Sep 17 15:50:07.581532 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:07.580623    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerID="40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9" exitCode=0
Sep 17 15:50:07.581532 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:07.580684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerDied","Data":"40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9"}
Sep 17 15:50:23.680325 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:23.680270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerStarted","Data":"c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099"}
Sep 17 15:50:23.682400 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:23.682371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerStarted","Data":"60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a"}
Sep 17 15:50:26.702582 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.702543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerStarted","Data":"d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb"}
Sep 17 15:50:26.703022 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.702624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:26.703022 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.702752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:26.704233 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.704208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:50:26.704861 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.704841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerStarted","Data":"2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49"}
Sep 17 15:50:26.704984 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.704970    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:26.705032 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.705008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:26.705922 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.705900    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:50:26.736032 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.735989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podStartSLOduration=1.906799521 podStartE2EDuration="24.735976889s" totalImagesPullingTime="19.02852216s" totalInitContainerRuntime="3.800655208s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:26.733683258 +0000 UTC m=+679.695982823" watchObservedRunningTime="2026-09-17 15:50:26.735976889 +0000 UTC m=+679.698276490"
Sep 17 15:50:26.762580 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:26.762523    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podStartSLOduration=2.966022966 podStartE2EDuration="24.762512249s" totalImagesPullingTime="18.024021152s" totalInitContainerRuntime="3.772468131s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:26.761522781 +0000 UTC m=+679.723822347" watchObservedRunningTime="2026-09-17 15:50:26.762512249 +0000 UTC m=+679.724811796"
Sep 17 15:50:27.709783 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:27.709739    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:50:27.710216 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:27.709955    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:50:32.716402 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:32.716371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:50:32.716835 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:32.716479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:50:32.717052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:32.717011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:50:32.717165 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:32.717009    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:50:42.717860 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:42.717820    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:50:42.718275 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:42.717834    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:50:52.717486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:52.717449    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:50:52.717891 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:50:52.717447    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:51:02.717046 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:02.716966    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:51:02.717046 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:02.717006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:51:12.717322 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:12.717279    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:51:12.717722 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:12.717279    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:51:22.717813 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:22.717768    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:51:22.717813 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:22.717801    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:51:32.717025 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:32.716979    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 15:51:32.717483 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:32.716979    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 15:51:41.534031 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:41.533998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:51:41.697233 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:41.697199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"]
Sep 17 15:51:41.697536 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:41.697507    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container" containerID="cri-o://60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a" gracePeriod=30
Sep 17 15:51:41.697651 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:41.697541    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kube-rbac-proxy" containerID="cri-o://2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49" gracePeriod=30
Sep 17 15:51:41.698436 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:41.698411    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:51:42.039000 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:42.038963    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerID="2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49" exitCode=2
Sep 17 15:51:42.039136 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:42.039032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerDied","Data":"2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49"}
Sep 17 15:51:42.710619 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:42.710571    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 15:51:45.538199 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.538177    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:51:45.601717 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.601684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-proxy-tls\") pod \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") "
Sep 17 15:51:45.601850 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.601752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-rbac-proxy-sar-config\") pod \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") "
Sep 17 15:51:45.601918 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.601875    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kserve-provision-location\") pod \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") "
Sep 17 15:51:45.601978 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.601927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-469qp\" (UniqueName: \"kubernetes.io/projected/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-api-access-469qp\") pod \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") "
Sep 17 15:51:45.601978 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.601969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-cabundle-cert\") pod \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\" (UID: \"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23\") "
Sep 17 15:51:45.602174 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.602057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" (UID: "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:51:45.602261 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.602180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" (UID: "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:51:45.602329 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.602258    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:45.602440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.602416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" (UID: "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:51:45.603914 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.603888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" (UID: "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:51:45.604007 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.603970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-api-access-469qp" (OuterVolumeSpecName: "kube-api-access-469qp") pod "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" (UID: "cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23"). InnerVolumeSpecName "kube-api-access-469qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:51:45.703052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.703023    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:45.703052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.703051    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:45.703163 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.703065    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469qp\" (UniqueName: \"kubernetes.io/projected/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-kube-api-access-469qp\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:45.703163 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:45.703077    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:46.060351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.060318    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerID="60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a" exitCode=0
Sep 17 15:51:46.060480 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.060384    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"
Sep 17 15:51:46.060480 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.060409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerDied","Data":"60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a"}
Sep 17 15:51:46.060480 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.060474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg" event={"ID":"cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23","Type":"ContainerDied","Data":"3ac0b7e77b4dc766b8a6909f1669aceb4017b14aa2dd489543176f3aa1e1fef6"}
Sep 17 15:51:46.060579 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.060504    2563 scope.go:117] "RemoveContainer" containerID="2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49"
Sep 17 15:51:46.070253 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.070220    2563 scope.go:117] "RemoveContainer" containerID="60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a"
Sep 17 15:51:46.078654 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.078638    2563 scope.go:117] "RemoveContainer" containerID="40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9"
Sep 17 15:51:46.086405 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.086381    2563 scope.go:117] "RemoveContainer" containerID="2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49"
Sep 17 15:51:46.086916 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:46.086889    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49\": container with ID starting with 2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49 not found: ID does not exist" containerID="2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49"
Sep 17 15:51:46.087055 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.086940    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49"} err="failed to get container status \"2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49\": rpc error: code = NotFound desc = could not find container \"2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49\": container with ID starting with 2dcc09d2c54e9ec62f4321c1a5a8316b60e2dbaa02425c6cb8fae1bcfba2af49 not found: ID does not exist"
Sep 17 15:51:46.087055 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.086963    2563 scope.go:117] "RemoveContainer" containerID="60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a"
Sep 17 15:51:46.087451 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:46.087413    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a\": container with ID starting with 60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a not found: ID does not exist" containerID="60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a"
Sep 17 15:51:46.087544 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.087460    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a"} err="failed to get container status \"60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a\": rpc error: code = NotFound desc = could not find container \"60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a\": container with ID starting with 60b0dea6cd5b652a2589709bfdfad19016762db378ea86641e412c0ed4c9dd7a not found: ID does not exist"
Sep 17 15:51:46.087544 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.087482    2563 scope.go:117] "RemoveContainer" containerID="40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9"
Sep 17 15:51:46.087858 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:46.087838    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9\": container with ID starting with 40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9 not found: ID does not exist" containerID="40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9"
Sep 17 15:51:46.087906 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.087863    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9"} err="failed to get container status \"40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9\": rpc error: code = NotFound desc = could not find container \"40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9\": container with ID starting with 40042be9a1403f503e6d561957fb26c8c0ea4cb1c38d3fd2294c016e840b00e9 not found: ID does not exist"
Sep 17 15:51:46.089126 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.089102    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"]
Sep 17 15:51:46.095167 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:46.095144    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-v6nkg"]
Sep 17 15:51:47.535469 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:47.535435    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" path="/var/lib/kubelet/pods/cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23/volumes"
Sep 17 15:51:52.115938 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.115899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"]
Sep 17 15:51:52.116416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.116127    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container" containerID="cri-o://c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099" gracePeriod=30
Sep 17 15:51:52.116416 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.116164    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kube-rbac-proxy" containerID="cri-o://d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb" gracePeriod=30
Sep 17 15:51:52.263813 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.263785    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"]
Sep 17 15:51:52.264142 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264129    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container"
Sep 17 15:51:52.264188 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264143    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container"
Sep 17 15:51:52.264188 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264171    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="storage-initializer"
Sep 17 15:51:52.264188 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264177    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="storage-initializer"
Sep 17 15:51:52.264188 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264185    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kube-rbac-proxy"
Sep 17 15:51:52.264342 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264191    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kube-rbac-proxy"
Sep 17 15:51:52.264342 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264238    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kserve-container"
Sep 17 15:51:52.264342 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.264271    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9ff287-f9dd-4b6a-a60b-94e08c6c5c23" containerName="kube-rbac-proxy"
Sep 17 15:51:52.266213 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.266198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.271676 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.271650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 15:51:52.293705 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.293680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"]
Sep 17 15:51:52.350730 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.350707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbcb\" (UniqueName: \"kubernetes.io/projected/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-api-access-jvbcb\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.350816 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.350753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.350870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.350850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.350914 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.350880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.350914 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.350909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452228 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452352 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbcb\" (UniqueName: \"kubernetes.io/projected/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-api-access-jvbcb\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452352 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452352 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452352 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.452967 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.453038 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.452970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.454673 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.454653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.474714 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.474684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbcb\" (UniqueName: \"kubernetes.io/projected/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-api-access-jvbcb\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-cn59f\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.576334 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.576313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:52.710474 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.710397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:51:52.862983 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:52.862798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"]
Sep 17 15:51:52.865534 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:51:52.865507    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9060f028_9d7c_4ef3_b9f3_e34f86d8b075.slice/crio-fbda428ee4db12f25f5148b84d780077baddb8111b8a119d97c2752273e66c7d WatchSource:0}: Error finding container fbda428ee4db12f25f5148b84d780077baddb8111b8a119d97c2752273e66c7d: Status 404 returned error can't find the container with id fbda428ee4db12f25f5148b84d780077baddb8111b8a119d97c2752273e66c7d
Sep 17 15:51:53.093934 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:53.093862    2563 generic.go:356] "Generic (PLEG): container finished" podID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerID="d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb" exitCode=2
Sep 17 15:51:53.094081 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:53.093942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerDied","Data":"d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb"}
Sep 17 15:51:53.095582 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:53.095559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerStarted","Data":"713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60"}
Sep 17 15:51:53.095708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:53.095622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerStarted","Data":"fbda428ee4db12f25f5148b84d780077baddb8111b8a119d97c2752273e66c7d"}
Sep 17 15:51:56.253763 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.253739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:51:56.283617 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.283595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-rbac-proxy-sar-config\") pod \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") "
Sep 17 15:51:56.283708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.283633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7de2c36c-d297-42a9-9f86-4a8c09e6e793-proxy-tls\") pod \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") "
Sep 17 15:51:56.283708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.283656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-cabundle-cert\") pod \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") "
Sep 17 15:51:56.283708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.283702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kserve-provision-location\") pod \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") "
Sep 17 15:51:56.283815 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.283756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9gs7\" (UniqueName: \"kubernetes.io/projected/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-api-access-m9gs7\") pod \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\" (UID: \"7de2c36c-d297-42a9-9f86-4a8c09e6e793\") "
Sep 17 15:51:56.284031 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.284004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7de2c36c-d297-42a9-9f86-4a8c09e6e793" (UID: "7de2c36c-d297-42a9-9f86-4a8c09e6e793"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:51:56.284108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.284002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7de2c36c-d297-42a9-9f86-4a8c09e6e793" (UID: "7de2c36c-d297-42a9-9f86-4a8c09e6e793"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:51:56.284108 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.284013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7de2c36c-d297-42a9-9f86-4a8c09e6e793" (UID: "7de2c36c-d297-42a9-9f86-4a8c09e6e793"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:51:56.285794 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.285768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-api-access-m9gs7" (OuterVolumeSpecName: "kube-api-access-m9gs7") pod "7de2c36c-d297-42a9-9f86-4a8c09e6e793" (UID: "7de2c36c-d297-42a9-9f86-4a8c09e6e793"). InnerVolumeSpecName "kube-api-access-m9gs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:51:56.285886 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.285831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de2c36c-d297-42a9-9f86-4a8c09e6e793-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7de2c36c-d297-42a9-9f86-4a8c09e6e793" (UID: "7de2c36c-d297-42a9-9f86-4a8c09e6e793"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:51:56.384575 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.384552    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:56.384575 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.384576    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gs7\" (UniqueName: \"kubernetes.io/projected/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-api-access-m9gs7\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:56.384701 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.384586    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:56.384701 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.384595    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7de2c36c-d297-42a9-9f86-4a8c09e6e793-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:56.384701 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:56.384606    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7de2c36c-d297-42a9-9f86-4a8c09e6e793-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:51:57.114290 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.114209    2563 generic.go:356] "Generic (PLEG): container finished" podID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerID="713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60" exitCode=0
Sep 17 15:51:57.114429 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.114286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerDied","Data":"713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60"}
Sep 17 15:51:57.116422 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.116400    2563 generic.go:356] "Generic (PLEG): container finished" podID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerID="c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099" exitCode=0
Sep 17 15:51:57.116528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.116451    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"
Sep 17 15:51:57.116528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.116454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerDied","Data":"c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099"}
Sep 17 15:51:57.116641 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.116572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr" event={"ID":"7de2c36c-d297-42a9-9f86-4a8c09e6e793","Type":"ContainerDied","Data":"5c8a27c7739a8d5f099a22b5a747d7433a985d6fb117a434b0ff25336b46dbb8"}
Sep 17 15:51:57.116688 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.116650    2563 scope.go:117] "RemoveContainer" containerID="d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb"
Sep 17 15:51:57.128870 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.127440    2563 scope.go:117] "RemoveContainer" containerID="c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099"
Sep 17 15:51:57.141372 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.141354    2563 scope.go:117] "RemoveContainer" containerID="74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5"
Sep 17 15:51:57.159124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.159100    2563 scope.go:117] "RemoveContainer" containerID="d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb"
Sep 17 15:51:57.159606 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:57.159578    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb\": container with ID starting with d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb not found: ID does not exist" containerID="d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb"
Sep 17 15:51:57.159702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.159617    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb"} err="failed to get container status \"d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb\": rpc error: code = NotFound desc = could not find container \"d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb\": container with ID starting with d8645470f519a78673b5fef780dfca16c96a0a783f69b1938977c7915a9a6cdb not found: ID does not exist"
Sep 17 15:51:57.159702 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.159641    2563 scope.go:117] "RemoveContainer" containerID="c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099"
Sep 17 15:51:57.159932 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:57.159914    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099\": container with ID starting with c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099 not found: ID does not exist" containerID="c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099"
Sep 17 15:51:57.160008 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.159937    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099"} err="failed to get container status \"c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099\": rpc error: code = NotFound desc = could not find container \"c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099\": container with ID starting with c06824134a71aa96a5bc1e44dfc00662e245431a3c8e7f1e06f20453134ef099 not found: ID does not exist"
Sep 17 15:51:57.160008 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.159952    2563 scope.go:117] "RemoveContainer" containerID="74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5"
Sep 17 15:51:57.160254 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:57.160223    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5\": container with ID starting with 74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5 not found: ID does not exist" containerID="74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5"
Sep 17 15:51:57.160347 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.160288    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5"} err="failed to get container status \"74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5\": rpc error: code = NotFound desc = could not find container \"74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5\": container with ID starting with 74a483124349483827264e08e113ee68f93a0e40903fc0647f0f06592e4662a5 not found: ID does not exist"
Sep 17 15:51:57.173723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.173703    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"]
Sep 17 15:51:57.187825 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.187807    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-g92sr"]
Sep 17 15:51:57.303160 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303134    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"]
Sep 17 15:51:57.303518 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="storage-initializer"
Sep 17 15:51:57.303567 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303519    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="storage-initializer"
Sep 17 15:51:57.303567 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container"
Sep 17 15:51:57.303567 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container"
Sep 17 15:51:57.303567 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303558    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kube-rbac-proxy"
Sep 17 15:51:57.303567 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303564    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kube-rbac-proxy"
Sep 17 15:51:57.303715 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303612    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kube-rbac-proxy"
Sep 17 15:51:57.303715 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.303624    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" containerName="kserve-container"
Sep 17 15:51:57.305859 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.305842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.309798 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.309775    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-edc89-kube-rbac-proxy-sar-config\""
Sep 17 15:51:57.309978 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.309963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-edc89-predictor-serving-cert\""
Sep 17 15:51:57.324120 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.322912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"]
Sep 17 15:51:57.392649 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.392622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10538fc5-d4ab-44e8-85cd-51b30f94123c-proxy-tls\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.392783 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.392670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.392783 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.392758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.392897 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.392811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10538fc5-d4ab-44e8-85cd-51b30f94123c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.392897 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.392861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sb6\" (UniqueName: \"kubernetes.io/projected/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-api-access-x5sb6\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494025 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10538fc5-d4ab-44e8-85cd-51b30f94123c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sb6\" (UniqueName: \"kubernetes.io/projected/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-api-access-x5sb6\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10538fc5-d4ab-44e8-85cd-51b30f94123c-proxy-tls\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10538fc5-d4ab-44e8-85cd-51b30f94123c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494777 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.494813 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.494792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.496554 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.496537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10538fc5-d4ab-44e8-85cd-51b30f94123c-proxy-tls\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.506145 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.506116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sb6\" (UniqueName: \"kubernetes.io/projected/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-api-access-x5sb6\") pod \"isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.534863 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.534836    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de2c36c-d297-42a9-9f86-4a8c09e6e793" path="/var/lib/kubelet/pods/7de2c36c-d297-42a9-9f86-4a8c09e6e793/volumes"
Sep 17 15:51:57.551953 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.551931    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"]
Sep 17 15:51:57.555585 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.555569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.559440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.559418    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-edc89-predictor-serving-cert\""
Sep 17 15:51:57.559440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.559434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-edc89-kube-rbac-proxy-sar-config\""
Sep 17 15:51:57.587610 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.587583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"]
Sep 17 15:51:57.595021 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.595001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.595104 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.595030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.595104 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.595048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbgj\" (UniqueName: \"kubernetes.io/projected/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-api-access-mnbgj\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.595206 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.595180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.595283 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.595264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad066c00-8135-4aa1-b331-b2e72df331f1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.616016 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.615993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:51:57.695776 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.695747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.695920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.695810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad066c00-8135-4aa1-b331-b2e72df331f1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.695920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.695860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.695920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.695881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.695920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.695899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbgj\" (UniqueName: \"kubernetes.io/projected/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-api-access-mnbgj\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.696141 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:57.695997    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-serving-cert: secret "isvc-xgboost-graph-raw-edc89-predictor-serving-cert" not found
Sep 17 15:51:57.696141 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:51:57.696065    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls podName:ad066c00-8135-4aa1-b331-b2e72df331f1 nodeName:}" failed. No retries permitted until 2026-09-17 15:51:58.19604426 +0000 UTC m=+771.158343800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls") pod "isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" (UID: "ad066c00-8135-4aa1-b331-b2e72df331f1") : secret "isvc-xgboost-graph-raw-edc89-predictor-serving-cert" not found
Sep 17 15:51:57.696373 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.696283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad066c00-8135-4aa1-b331-b2e72df331f1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.696647 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.696624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.696798 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.696776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.713539 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.713515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbgj\" (UniqueName: \"kubernetes.io/projected/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-api-access-mnbgj\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:57.902868 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:57.902844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"]
Sep 17 15:51:57.907645 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:51:57.907624    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10538fc5_d4ab_44e8_85cd_51b30f94123c.slice/crio-965a182bf0ac3119ec26f0cd580ad51dde4fd8526886a36a7a8137aee7061735 WatchSource:0}: Error finding container 965a182bf0ac3119ec26f0cd580ad51dde4fd8526886a36a7a8137aee7061735: Status 404 returned error can't find the container with id 965a182bf0ac3119ec26f0cd580ad51dde4fd8526886a36a7a8137aee7061735
Sep 17 15:51:58.125201 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.125167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerStarted","Data":"968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14"}
Sep 17 15:51:58.125414 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.125261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerStarted","Data":"43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931"}
Sep 17 15:51:58.125709 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.125625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:58.125709 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.125676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:51:58.126917 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.126889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerStarted","Data":"3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b"}
Sep 17 15:51:58.127023 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.126945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerStarted","Data":"965a182bf0ac3119ec26f0cd580ad51dde4fd8526886a36a7a8137aee7061735"}
Sep 17 15:51:58.127023 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.126996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:51:58.159986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.159935    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podStartSLOduration=2.228240697 podStartE2EDuration="6.159915984s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.931675287s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:51:58.157548406 +0000 UTC m=+771.119847980" watchObservedRunningTime="2026-09-17 15:51:58.159915984 +0000 UTC m=+771.122215533"
Sep 17 15:51:58.200335 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.200279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:58.202719 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.202693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:58.465632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.465558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:51:58.759013 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:58.758960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"]
Sep 17 15:51:58.761321 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:51:58.761289    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad066c00_8135_4aa1_b331_b2e72df331f1.slice/crio-13db3ce48939607504b9e7819666e2b76bebeadcbad7fd3af6518d2e88800d89 WatchSource:0}: Error finding container 13db3ce48939607504b9e7819666e2b76bebeadcbad7fd3af6518d2e88800d89: Status 404 returned error can't find the container with id 13db3ce48939607504b9e7819666e2b76bebeadcbad7fd3af6518d2e88800d89
Sep 17 15:51:59.134331 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:59.134290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerStarted","Data":"0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298"}
Sep 17 15:51:59.134489 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:59.134371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerStarted","Data":"13db3ce48939607504b9e7819666e2b76bebeadcbad7fd3af6518d2e88800d89"}
Sep 17 15:51:59.134805 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:51:59.134769    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:02.150060 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:02.150022    2563 generic.go:356] "Generic (PLEG): container finished" podID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerID="3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b" exitCode=0
Sep 17 15:52:02.150449 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:02.150096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerDied","Data":"3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b"}
Sep 17 15:52:03.158576 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.158542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerStarted","Data":"c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1"}
Sep 17 15:52:03.158980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.158613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerStarted","Data":"18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b"}
Sep 17 15:52:03.159067 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.159026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:52:03.159199 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.159178    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:52:03.160813 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.160781    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:03.160977 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.160950    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerID="0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298" exitCode=0
Sep 17 15:52:03.161052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.161021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerDied","Data":"0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298"}
Sep 17 15:52:03.195533 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:03.195488    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podStartSLOduration=2.434522547 podStartE2EDuration="6.195471457s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.76094891s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:52:03.193584077 +0000 UTC m=+776.155883651" watchObservedRunningTime="2026-09-17 15:52:03.195471457 +0000 UTC m=+776.157771005"
Sep 17 15:52:04.140334 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:04.140301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:52:04.140929 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:04.140896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:04.168124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:04.168084    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:09.173083 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:09.173048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:52:09.173775 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:09.173738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:14.141351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:14.141312    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:19.174509 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:19.174471    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:23.268512 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:23.268479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerStarted","Data":"21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2"}
Sep 17 15:52:23.268904 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:23.268545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerStarted","Data":"1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59"}
Sep 17 15:52:23.268904 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:23.268844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:52:23.312885 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:23.312838    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podStartSLOduration=3.042157443 podStartE2EDuration="26.312826056s" totalImagesPullingTime="19.499478928s" totalInitContainerRuntime="3.771189685s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:52:23.312742699 +0000 UTC m=+796.275042256" watchObservedRunningTime="2026-09-17 15:52:23.312826056 +0000 UTC m=+796.275125602"
Sep 17 15:52:24.140864 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:24.140829    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:24.272982 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:24.272952    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:52:24.274257 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:24.274215    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:52:25.278052 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:25.278009    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:52:29.173791 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:29.173750    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:30.282439 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:30.282365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:52:30.283028 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:30.282999    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:52:34.140872 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:34.140825    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:39.173912 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:39.173872    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:40.283612 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:40.283568    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:52:44.141138 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:44.141099    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:49.173920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:49.173877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:52:50.283479 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:50.283437    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:52:54.141620 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:54.141580    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:52:59.173785 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:52:59.173747    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:53:00.283560 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:00.283523    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:53:04.140933 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:04.140891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:53:09.173927 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:09.173884    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:53:10.283710 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:10.283671    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:53:14.142204 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:14.142176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:53:19.174401 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:19.174372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:53:20.282946 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:20.282908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:53:22.440094 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:22.440055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"]
Sep 17 15:53:22.440486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:22.440319    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" containerID="cri-o://43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931" gracePeriod=30
Sep 17 15:53:22.440486 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:22.440351    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kube-rbac-proxy" containerID="cri-o://968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14" gracePeriod=30
Sep 17 15:53:23.558643 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:23.558603    2563 generic.go:356] "Generic (PLEG): container finished" podID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerID="968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14" exitCode=2
Sep 17 15:53:23.559009 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:23.558668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerDied","Data":"968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14"}
Sep 17 15:53:24.135099 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:24.135061    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:53:24.141480 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:24.141444    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:53:26.280124 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.280103    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:53:26.408580 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.408549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-proxy-tls\") pod \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") "
Sep 17 15:53:26.408708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.408606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kserve-provision-location\") pod \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") "
Sep 17 15:53:26.408708 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.408638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvbcb\" (UniqueName: \"kubernetes.io/projected/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-api-access-jvbcb\") pod \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") "
Sep 17 15:53:26.408788 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.408720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-cabundle-cert\") pod \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") "
Sep 17 15:53:26.408788 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.408747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-rbac-proxy-sar-config\") pod \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\" (UID: \"9060f028-9d7c-4ef3-b9f3-e34f86d8b075\") "
Sep 17 15:53:26.409044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.408944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9060f028-9d7c-4ef3-b9f3-e34f86d8b075" (UID: "9060f028-9d7c-4ef3-b9f3-e34f86d8b075"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:26.409175 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.409117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9060f028-9d7c-4ef3-b9f3-e34f86d8b075" (UID: "9060f028-9d7c-4ef3-b9f3-e34f86d8b075"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:26.409175 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.409142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9060f028-9d7c-4ef3-b9f3-e34f86d8b075" (UID: "9060f028-9d7c-4ef3-b9f3-e34f86d8b075"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:26.410626 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.410606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9060f028-9d7c-4ef3-b9f3-e34f86d8b075" (UID: "9060f028-9d7c-4ef3-b9f3-e34f86d8b075"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:26.410816 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.410793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-api-access-jvbcb" (OuterVolumeSpecName: "kube-api-access-jvbcb") pod "9060f028-9d7c-4ef3-b9f3-e34f86d8b075" (UID: "9060f028-9d7c-4ef3-b9f3-e34f86d8b075"). InnerVolumeSpecName "kube-api-access-jvbcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:26.509871 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.509841    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbcb\" (UniqueName: \"kubernetes.io/projected/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-api-access-jvbcb\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.509871 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.509869    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.509871 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.509879    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.510034 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.509887    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.510034 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.509897    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9060f028-9d7c-4ef3-b9f3-e34f86d8b075-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.574654 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.574626    2563 generic.go:356] "Generic (PLEG): container finished" podID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerID="43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931" exitCode=0
Sep 17 15:53:26.574749 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.574690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"
Sep 17 15:53:26.574749 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.574717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerDied","Data":"43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931"}
Sep 17 15:53:26.574823 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.574778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f" event={"ID":"9060f028-9d7c-4ef3-b9f3-e34f86d8b075","Type":"ContainerDied","Data":"fbda428ee4db12f25f5148b84d780077baddb8111b8a119d97c2752273e66c7d"}
Sep 17 15:53:26.574823 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.574809    2563 scope.go:117] "RemoveContainer" containerID="968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14"
Sep 17 15:53:26.586033 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.586015    2563 scope.go:117] "RemoveContainer" containerID="43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931"
Sep 17 15:53:26.593879 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.593860    2563 scope.go:117] "RemoveContainer" containerID="713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60"
Sep 17 15:53:26.601195 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.601178    2563 scope.go:117] "RemoveContainer" containerID="968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14"
Sep 17 15:53:26.601586 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:26.601567    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14\": container with ID starting with 968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14 not found: ID does not exist" containerID="968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14"
Sep 17 15:53:26.601649 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.601595    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14"} err="failed to get container status \"968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14\": rpc error: code = NotFound desc = could not find container \"968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14\": container with ID starting with 968c700a75c6e8bbe85f4bf28952b96151b8b67068cc4692e3dc82bfd5347a14 not found: ID does not exist"
Sep 17 15:53:26.601649 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.601613    2563 scope.go:117] "RemoveContainer" containerID="43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931"
Sep 17 15:53:26.601962 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:26.601939    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931\": container with ID starting with 43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931 not found: ID does not exist" containerID="43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931"
Sep 17 15:53:26.602015 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.601971    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931"} err="failed to get container status \"43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931\": rpc error: code = NotFound desc = could not find container \"43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931\": container with ID starting with 43acb4884b64c4e35835636bd88fe1cdca1159209f797114ec5b4ebed5f8f931 not found: ID does not exist"
Sep 17 15:53:26.602015 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.601991    2563 scope.go:117] "RemoveContainer" containerID="713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60"
Sep 17 15:53:26.602292 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:26.602273    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60\": container with ID starting with 713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60 not found: ID does not exist" containerID="713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60"
Sep 17 15:53:26.602345 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.602297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60"} err="failed to get container status \"713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60\": rpc error: code = NotFound desc = could not find container \"713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60\": container with ID starting with 713e5753b50fa58d92481b99767803bac2f504806260999381922e27bd9f3b60 not found: ID does not exist"
Sep 17 15:53:26.607958 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.607936    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"]
Sep 17 15:53:26.617605 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:26.617585    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-cn59f"]
Sep 17 15:53:27.535206 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.535173    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" path="/var/lib/kubelet/pods/9060f028-9d7c-4ef3-b9f3-e34f86d8b075/volumes"
Sep 17 15:53:27.708321 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"]
Sep 17 15:53:27.708704 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container"
Sep 17 15:53:27.708765 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708705    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container"
Sep 17 15:53:27.708765 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708719    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="storage-initializer"
Sep 17 15:53:27.708765 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708725    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="storage-initializer"
Sep 17 15:53:27.708765 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708746    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kube-rbac-proxy"
Sep 17 15:53:27.708765 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708751    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kube-rbac-proxy"
Sep 17 15:53:27.708983 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708808    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kserve-container"
Sep 17 15:53:27.708983 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.708815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9060f028-9d7c-4ef3-b9f3-e34f86d8b075" containerName="kube-rbac-proxy"
Sep 17 15:53:27.712193 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.712168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.715307 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.715288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 15:53:27.715845 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.715817    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 15:53:27.731259 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.731219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"]
Sep 17 15:53:27.821169 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.821113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.821169 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.821157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d8f634-6511-4e87-a35a-9cd55a4d8391-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.821332 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.821268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.821332 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.821293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.821332 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.821330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6p5\" (UniqueName: \"kubernetes.io/projected/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-api-access-5k6p5\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.922467 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.922446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.922549 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.922486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d8f634-6511-4e87-a35a-9cd55a4d8391-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.922549 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.922533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.922627 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.922560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.922627 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.922601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6p5\" (UniqueName: \"kubernetes.io/projected/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-api-access-5k6p5\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.922986 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.922953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.923106 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.923088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.923207 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.923193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.924985 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.924964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d8f634-6511-4e87-a35a-9cd55a4d8391-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:27.939594 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:27.939566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6p5\" (UniqueName: \"kubernetes.io/projected/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-api-access-5k6p5\") pod \"isvc-canary-stop-predictor-556d599d48-whrrt\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:28.023827 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:28.023803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:28.306627 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:28.306604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"]
Sep 17 15:53:28.309278 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:53:28.309236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d8f634_6511_4e87_a35a_9cd55a4d8391.slice/crio-925d73915666b9d3ff0ec9cac2f2297a894475be795cd64f7022c333db54603a WatchSource:0}: Error finding container 925d73915666b9d3ff0ec9cac2f2297a894475be795cd64f7022c333db54603a: Status 404 returned error can't find the container with id 925d73915666b9d3ff0ec9cac2f2297a894475be795cd64f7022c333db54603a
Sep 17 15:53:28.586481 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:28.586400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerStarted","Data":"b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9"}
Sep 17 15:53:28.586801 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:28.586510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerStarted","Data":"925d73915666b9d3ff0ec9cac2f2297a894475be795cd64f7022c333db54603a"}
Sep 17 15:53:30.283441 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:30.283409    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:53:32.607630 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:32.607595    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerID="b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9" exitCode=0
Sep 17 15:53:32.607992 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:32.607669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerDied","Data":"b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9"}
Sep 17 15:53:33.614903 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:33.614868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerStarted","Data":"0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579"}
Sep 17 15:53:33.615329 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:33.614925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerStarted","Data":"c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259"}
Sep 17 15:53:33.615329 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:33.615224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:33.615329 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:33.615313    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:33.616568 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:33.616537    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:53:34.619773 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:34.619729    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:53:39.624330 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:39.624297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:53:39.624802 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:39.624776    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:53:39.659780 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:39.659736    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podStartSLOduration=8.983779814 podStartE2EDuration="12.6597225s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675942686s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:33.650069617 +0000 UTC m=+866.612369173" watchObservedRunningTime="2026-09-17 15:53:39.6597225 +0000 UTC m=+872.622022081"
Sep 17 15:53:47.733646 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:47.733619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"]
Sep 17 15:53:47.734068 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:47.733852    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" containerID="cri-o://18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b" gracePeriod=30
Sep 17 15:53:47.734068 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:47.733916    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kube-rbac-proxy" containerID="cri-o://c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1" gracePeriod=30
Sep 17 15:53:48.035809 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.035732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"]
Sep 17 15:53:48.035980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.035955    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" containerID="cri-o://1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59" gracePeriod=30
Sep 17 15:53:48.036040 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.035997    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kube-rbac-proxy" containerID="cri-o://21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2" gracePeriod=30
Sep 17 15:53:48.096535 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.096505    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"]
Sep 17 15:53:48.099418 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.099401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.103228 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.103198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-883e7-kube-rbac-proxy-sar-config\""
Sep 17 15:53:48.103346 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.103262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-883e7-predictor-serving-cert\""
Sep 17 15:53:48.125475 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.125452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"]
Sep 17 15:53:48.178723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.178696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.178818 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.178731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.178818 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.178763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.178818 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.178785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmxk\" (UniqueName: \"kubernetes.io/projected/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-api-access-dsmxk\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.178935 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.178839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.279594 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.279563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.279695 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.279604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.279695 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.279630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmxk\" (UniqueName: \"kubernetes.io/projected/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-api-access-dsmxk\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.279695 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.279663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.279850 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.279782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.280017 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.279992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.280380 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.280360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.280425 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.280386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.281859 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.281841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.305423 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.305362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmxk\" (UniqueName: \"kubernetes.io/projected/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-api-access-dsmxk\") pod \"isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.409261 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.409218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:48.688450 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.688420    2563 generic.go:356] "Generic (PLEG): container finished" podID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerID="c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1" exitCode=2
Sep 17 15:53:48.688593 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.688498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerDied","Data":"c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1"}
Sep 17 15:53:48.690642 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.690615    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerID="21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2" exitCode=2
Sep 17 15:53:48.690766 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.690657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerDied","Data":"21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2"}
Sep 17 15:53:48.696690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:48.696671    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"]
Sep 17 15:53:48.698920 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:53:48.698899    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad0347c_d42b_4509_a0ec_4e9b2dc34ca0.slice/crio-1638a9e771bf4dfb8fa6668c3bec7b7c4fe8f0783791b9ad4a6fa42b94a4e427 WatchSource:0}: Error finding container 1638a9e771bf4dfb8fa6668c3bec7b7c4fe8f0783791b9ad4a6fa42b94a4e427: Status 404 returned error can't find the container with id 1638a9e771bf4dfb8fa6668c3bec7b7c4fe8f0783791b9ad4a6fa42b94a4e427
Sep 17 15:53:49.168344 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:49.168307    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 15:53:49.173901 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:49.173874    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:53:49.625588 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:49.625555    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:53:49.696471 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:49.696441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerStarted","Data":"4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b"}
Sep 17 15:53:49.696622 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:49.696509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerStarted","Data":"1638a9e771bf4dfb8fa6668c3bec7b7c4fe8f0783791b9ad4a6fa42b94a4e427"}
Sep 17 15:53:50.279258 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:50.279197    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 15:53:50.283148 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:50.283120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:53:51.981494 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:51.981470    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:53:52.015728 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.015706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10538fc5-d4ab-44e8-85cd-51b30f94123c-kserve-provision-location\") pod \"10538fc5-d4ab-44e8-85cd-51b30f94123c\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") "
Sep 17 15:53:52.015841 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.015748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-cabundle-cert\") pod \"10538fc5-d4ab-44e8-85cd-51b30f94123c\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") "
Sep 17 15:53:52.015841 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.015798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10538fc5-d4ab-44e8-85cd-51b30f94123c-proxy-tls\") pod \"10538fc5-d4ab-44e8-85cd-51b30f94123c\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") "
Sep 17 15:53:52.015841 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.015815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5sb6\" (UniqueName: \"kubernetes.io/projected/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-api-access-x5sb6\") pod \"10538fc5-d4ab-44e8-85cd-51b30f94123c\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") "
Sep 17 15:53:52.016010 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.015850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-rbac-proxy-sar-config\") pod \"10538fc5-d4ab-44e8-85cd-51b30f94123c\" (UID: \"10538fc5-d4ab-44e8-85cd-51b30f94123c\") "
Sep 17 15:53:52.016079 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.016050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10538fc5-d4ab-44e8-85cd-51b30f94123c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10538fc5-d4ab-44e8-85cd-51b30f94123c" (UID: "10538fc5-d4ab-44e8-85cd-51b30f94123c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:52.016186 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.016165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10538fc5-d4ab-44e8-85cd-51b30f94123c" (UID: "10538fc5-d4ab-44e8-85cd-51b30f94123c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:52.016272 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.016222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10538fc5-d4ab-44e8-85cd-51b30f94123c" (UID: "10538fc5-d4ab-44e8-85cd-51b30f94123c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:52.017830 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.017801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10538fc5-d4ab-44e8-85cd-51b30f94123c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10538fc5-d4ab-44e8-85cd-51b30f94123c" (UID: "10538fc5-d4ab-44e8-85cd-51b30f94123c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:52.017920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.017872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-api-access-x5sb6" (OuterVolumeSpecName: "kube-api-access-x5sb6") pod "10538fc5-d4ab-44e8-85cd-51b30f94123c" (UID: "10538fc5-d4ab-44e8-85cd-51b30f94123c"). InnerVolumeSpecName "kube-api-access-x5sb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:52.117543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.117489    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10538fc5-d4ab-44e8-85cd-51b30f94123c-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.117543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.117513    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5sb6\" (UniqueName: \"kubernetes.io/projected/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-api-access-x5sb6\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.117543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.117529    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.117691 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.117543    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10538fc5-d4ab-44e8-85cd-51b30f94123c-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.117691 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.117556    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10538fc5-d4ab-44e8-85cd-51b30f94123c-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.261995 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.261973    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:53:52.319690 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.319658    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnbgj\" (UniqueName: \"kubernetes.io/projected/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-api-access-mnbgj\") pod \"ad066c00-8135-4aa1-b331-b2e72df331f1\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") "
Sep 17 15:53:52.319837 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.319721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad066c00-8135-4aa1-b331-b2e72df331f1-kserve-provision-location\") pod \"ad066c00-8135-4aa1-b331-b2e72df331f1\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") "
Sep 17 15:53:52.319837 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.319797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-cabundle-cert\") pod \"ad066c00-8135-4aa1-b331-b2e72df331f1\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") "
Sep 17 15:53:52.319941 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.319854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls\") pod \"ad066c00-8135-4aa1-b331-b2e72df331f1\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") "
Sep 17 15:53:52.319941 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.319907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-rbac-proxy-sar-config\") pod \"ad066c00-8135-4aa1-b331-b2e72df331f1\" (UID: \"ad066c00-8135-4aa1-b331-b2e72df331f1\") "
Sep 17 15:53:52.320055 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.319998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad066c00-8135-4aa1-b331-b2e72df331f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad066c00-8135-4aa1-b331-b2e72df331f1" (UID: "ad066c00-8135-4aa1-b331-b2e72df331f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:52.320199 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.320174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad066c00-8135-4aa1-b331-b2e72df331f1" (UID: "ad066c00-8135-4aa1-b331-b2e72df331f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:52.320370 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.320186    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad066c00-8135-4aa1-b331-b2e72df331f1-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.320432 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.320391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad066c00-8135-4aa1-b331-b2e72df331f1" (UID: "ad066c00-8135-4aa1-b331-b2e72df331f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:52.321853 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.321832    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad066c00-8135-4aa1-b331-b2e72df331f1" (UID: "ad066c00-8135-4aa1-b331-b2e72df331f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:52.322040 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.322015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-api-access-mnbgj" (OuterVolumeSpecName: "kube-api-access-mnbgj") pod "ad066c00-8135-4aa1-b331-b2e72df331f1" (UID: "ad066c00-8135-4aa1-b331-b2e72df331f1"). InnerVolumeSpecName "kube-api-access-mnbgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:52.421044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.421010    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.421044 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.421044    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbgj\" (UniqueName: \"kubernetes.io/projected/ad066c00-8135-4aa1-b331-b2e72df331f1-kube-api-access-mnbgj\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.421220 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.421061    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad066c00-8135-4aa1-b331-b2e72df331f1-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.421220 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.421076    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad066c00-8135-4aa1-b331-b2e72df331f1-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:53:52.712922 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.712845    2563 generic.go:356] "Generic (PLEG): container finished" podID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerID="4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b" exitCode=0
Sep 17 15:53:52.713051 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.712928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerDied","Data":"4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b"}
Sep 17 15:53:52.715012 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.714986    2563 generic.go:356] "Generic (PLEG): container finished" podID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerID="18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b" exitCode=0
Sep 17 15:53:52.715116 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.715045    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"
Sep 17 15:53:52.715116 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.715075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerDied","Data":"18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b"}
Sep 17 15:53:52.715281 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.715146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb" event={"ID":"10538fc5-d4ab-44e8-85cd-51b30f94123c","Type":"ContainerDied","Data":"965a182bf0ac3119ec26f0cd580ad51dde4fd8526886a36a7a8137aee7061735"}
Sep 17 15:53:52.715281 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.715187    2563 scope.go:117] "RemoveContainer" containerID="c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1"
Sep 17 15:53:52.717316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.717222    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerID="1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59" exitCode=0
Sep 17 15:53:52.717316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.717267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerDied","Data":"1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59"}
Sep 17 15:53:52.717316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.717286    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"
Sep 17 15:53:52.717316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.717309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw" event={"ID":"ad066c00-8135-4aa1-b331-b2e72df331f1","Type":"ContainerDied","Data":"13db3ce48939607504b9e7819666e2b76bebeadcbad7fd3af6518d2e88800d89"}
Sep 17 15:53:52.726115 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.726099    2563 scope.go:117] "RemoveContainer" containerID="18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b"
Sep 17 15:53:52.733772 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.733754    2563 scope.go:117] "RemoveContainer" containerID="3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b"
Sep 17 15:53:52.748076 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.748045    2563 scope.go:117] "RemoveContainer" containerID="c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1"
Sep 17 15:53:52.748438 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:52.748412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1\": container with ID starting with c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1 not found: ID does not exist" containerID="c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1"
Sep 17 15:53:52.748517 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.748450    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1"} err="failed to get container status \"c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1\": rpc error: code = NotFound desc = could not find container \"c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1\": container with ID starting with c03adad819d9067555e808732439e9734c1b1d50f5973bd8a6e95eeb076090d1 not found: ID does not exist"
Sep 17 15:53:52.748517 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.748478    2563 scope.go:117] "RemoveContainer" containerID="18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b"
Sep 17 15:53:52.748822 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:52.748791    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b\": container with ID starting with 18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b not found: ID does not exist" containerID="18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b"
Sep 17 15:53:52.748918 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.748816    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b"} err="failed to get container status \"18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b\": rpc error: code = NotFound desc = could not find container \"18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b\": container with ID starting with 18e00d9ff52b9447aa8b12bcb5a5c4727553eb8701f2c040864d498b4781844b not found: ID does not exist"
Sep 17 15:53:52.748918 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.748837    2563 scope.go:117] "RemoveContainer" containerID="3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b"
Sep 17 15:53:52.749277 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:52.749236    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b\": container with ID starting with 3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b not found: ID does not exist" containerID="3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b"
Sep 17 15:53:52.749389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.749287    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b"} err="failed to get container status \"3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b\": rpc error: code = NotFound desc = could not find container \"3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b\": container with ID starting with 3091711970d3859cbce3b8b1b1ff906f9a4aa52b21401a8db9f914507da9079b not found: ID does not exist"
Sep 17 15:53:52.749389 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.749310    2563 scope.go:117] "RemoveContainer" containerID="21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2"
Sep 17 15:53:52.772734 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.772712    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"]
Sep 17 15:53:52.776780 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.776757    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-edc89-predictor-6c4df5f8c7-hjnfb"]
Sep 17 15:53:52.787895 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.787875    2563 scope.go:117] "RemoveContainer" containerID="1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59"
Sep 17 15:53:52.795487 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.795454    2563 scope.go:117] "RemoveContainer" containerID="0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298"
Sep 17 15:53:52.797920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.797899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"]
Sep 17 15:53:52.803832 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.803811    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-edc89-predictor-74f46cd46f-mfsmw"]
Sep 17 15:53:52.804890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.804875    2563 scope.go:117] "RemoveContainer" containerID="21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2"
Sep 17 15:53:52.805179 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:52.805159    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2\": container with ID starting with 21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2 not found: ID does not exist" containerID="21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2"
Sep 17 15:53:52.805309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.805182    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2"} err="failed to get container status \"21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2\": rpc error: code = NotFound desc = could not find container \"21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2\": container with ID starting with 21758d444e38ce613545ad47a20e6ef5eed4fe133430209cc1587ad7917218a2 not found: ID does not exist"
Sep 17 15:53:52.805309 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.805203    2563 scope.go:117] "RemoveContainer" containerID="1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59"
Sep 17 15:53:52.805522 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:52.805500    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59\": container with ID starting with 1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59 not found: ID does not exist" containerID="1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59"
Sep 17 15:53:52.805583 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.805531    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59"} err="failed to get container status \"1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59\": rpc error: code = NotFound desc = could not find container \"1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59\": container with ID starting with 1925775567e7a632e329b531ea9d0ebfa1fd163be47ea9a3bcbacaa78c0cbd59 not found: ID does not exist"
Sep 17 15:53:52.805583 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.805553    2563 scope.go:117] "RemoveContainer" containerID="0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298"
Sep 17 15:53:52.805880 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:53:52.805859    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298\": container with ID starting with 0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298 not found: ID does not exist" containerID="0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298"
Sep 17 15:53:52.805956 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:52.805884    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298"} err="failed to get container status \"0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298\": rpc error: code = NotFound desc = could not find container \"0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298\": container with ID starting with 0bf0513704b309487aa80ee5d5dca1bb421c0f93e6e15bb70a85017fa162b298 not found: ID does not exist"
Sep 17 15:53:53.534991 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:53.534955    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" path="/var/lib/kubelet/pods/10538fc5-d4ab-44e8-85cd-51b30f94123c/volumes"
Sep 17 15:53:53.535464 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:53.535449    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" path="/var/lib/kubelet/pods/ad066c00-8135-4aa1-b331-b2e72df331f1/volumes"
Sep 17 15:53:53.725233 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:53.725201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerStarted","Data":"2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475"}
Sep 17 15:53:53.725394 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:53.725279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerStarted","Data":"1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98"}
Sep 17 15:53:53.725484 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:53.725461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:53.755716 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:53.755663    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podStartSLOduration=1.961898932 podStartE2EDuration="5.755645597s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793746665s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:53.754770069 +0000 UTC m=+886.717069717" watchObservedRunningTime="2026-09-17 15:53:53.755645597 +0000 UTC m=+886.717945145"
Sep 17 15:53:54.729701 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:54.729673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:53:54.730873 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:54.730848    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:53:55.733940 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:55.733904    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:53:59.625156 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:53:59.625109    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:54:00.738803 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:00.738773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:54:00.739399 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:00.739368    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:54:09.624760 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:09.624721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:54:10.739749 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:10.739712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:54:19.625763 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:19.625720    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:54:20.739395 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:20.739358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:54:29.625295 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:29.625229    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:54:30.739335 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:30.739289    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:54:39.625720 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:39.625678    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:54:40.739312 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:40.739272    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:54:49.625682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:49.625655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:54:50.740217 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:50.740174    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:54:57.538506 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:57.538467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"]
Sep 17 15:54:57.538959 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:57.538703    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" containerID="cri-o://c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259" gracePeriod=30
Sep 17 15:54:57.538959 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:57.538722    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kube-rbac-proxy" containerID="cri-o://0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579" gracePeriod=30
Sep 17 15:54:58.023750 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:58.023718    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerID="0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579" exitCode=2
Sep 17 15:54:58.023873 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:58.023801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerDied","Data":"0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579"}
Sep 17 15:54:59.619988 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:59.619949    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 15:54:59.625358 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:54:59.625330    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 15:55:00.744967 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:00.744935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:55:01.378414 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.378390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:55:01.449948 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.449886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kserve-provision-location\") pod \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") "
Sep 17 15:55:01.449948 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.449938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d8f634-6511-4e87-a35a-9cd55a4d8391-proxy-tls\") pod \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") "
Sep 17 15:55:01.450091 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.449970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-rbac-proxy-sar-config\") pod \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") "
Sep 17 15:55:01.450091 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.449986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k6p5\" (UniqueName: \"kubernetes.io/projected/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-api-access-5k6p5\") pod \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") "
Sep 17 15:55:01.450091 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.450019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-cabundle-cert\") pod \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\" (UID: \"b8d8f634-6511-4e87-a35a-9cd55a4d8391\") "
Sep 17 15:55:01.450226 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.450198    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8d8f634-6511-4e87-a35a-9cd55a4d8391" (UID: "b8d8f634-6511-4e87-a35a-9cd55a4d8391"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:01.450310 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.450288    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:01.450356 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.450328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8d8f634-6511-4e87-a35a-9cd55a4d8391" (UID: "b8d8f634-6511-4e87-a35a-9cd55a4d8391"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:01.450494 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.450468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8d8f634-6511-4e87-a35a-9cd55a4d8391" (UID: "b8d8f634-6511-4e87-a35a-9cd55a4d8391"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:01.452062 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.452041    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-api-access-5k6p5" (OuterVolumeSpecName: "kube-api-access-5k6p5") pod "b8d8f634-6511-4e87-a35a-9cd55a4d8391" (UID: "b8d8f634-6511-4e87-a35a-9cd55a4d8391"). InnerVolumeSpecName "kube-api-access-5k6p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:01.452146 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.452073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d8f634-6511-4e87-a35a-9cd55a4d8391-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8d8f634-6511-4e87-a35a-9cd55a4d8391" (UID: "b8d8f634-6511-4e87-a35a-9cd55a4d8391"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:01.551004 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.550976    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d8f634-6511-4e87-a35a-9cd55a4d8391-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:01.551004 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.551004    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:01.551113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.551013    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6p5\" (UniqueName: \"kubernetes.io/projected/b8d8f634-6511-4e87-a35a-9cd55a4d8391-kube-api-access-5k6p5\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:01.551113 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:01.551023    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d8f634-6511-4e87-a35a-9cd55a4d8391-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:02.044000 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.043968    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerID="c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259" exitCode=0
Sep 17 15:55:02.044351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.044030    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"
Sep 17 15:55:02.044351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.044062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerDied","Data":"c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259"}
Sep 17 15:55:02.044351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.044131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt" event={"ID":"b8d8f634-6511-4e87-a35a-9cd55a4d8391","Type":"ContainerDied","Data":"925d73915666b9d3ff0ec9cac2f2297a894475be795cd64f7022c333db54603a"}
Sep 17 15:55:02.044351 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.044165    2563 scope.go:117] "RemoveContainer" containerID="0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579"
Sep 17 15:55:02.053502 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.053482    2563 scope.go:117] "RemoveContainer" containerID="c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259"
Sep 17 15:55:02.061572 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.061550    2563 scope.go:117] "RemoveContainer" containerID="b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9"
Sep 17 15:55:02.069584 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.069566    2563 scope.go:117] "RemoveContainer" containerID="0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579"
Sep 17 15:55:02.069875 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:55:02.069857    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579\": container with ID starting with 0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579 not found: ID does not exist" containerID="0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579"
Sep 17 15:55:02.069920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.069883    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579"} err="failed to get container status \"0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579\": rpc error: code = NotFound desc = could not find container \"0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579\": container with ID starting with 0c222140fa9a3d65a122833ecb5d70fbec5d2bc93eb14f65efbfa735190be579 not found: ID does not exist"
Sep 17 15:55:02.069920 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.069902    2563 scope.go:117] "RemoveContainer" containerID="c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259"
Sep 17 15:55:02.070166 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:55:02.070145    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259\": container with ID starting with c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259 not found: ID does not exist" containerID="c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259"
Sep 17 15:55:02.070203 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.070174    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259"} err="failed to get container status \"c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259\": rpc error: code = NotFound desc = could not find container \"c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259\": container with ID starting with c43eb954d8fef0971e1d54c271f62cffd57a05906484900f7f3b1bb7c713f259 not found: ID does not exist"
Sep 17 15:55:02.070203 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.070191    2563 scope.go:117] "RemoveContainer" containerID="b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9"
Sep 17 15:55:02.070440 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:55:02.070423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9\": container with ID starting with b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9 not found: ID does not exist" containerID="b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9"
Sep 17 15:55:02.070491 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.070442    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9"} err="failed to get container status \"b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9\": rpc error: code = NotFound desc = could not find container \"b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9\": container with ID starting with b3ae5879db8e779cbb9e8bdaddae1a713ef8cce624db68163a5edced9d681fa9 not found: ID does not exist"
Sep 17 15:55:02.079152 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.079127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"]
Sep 17 15:55:02.083497 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:02.083474    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-whrrt"]
Sep 17 15:55:03.535593 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:03.535550    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" path="/var/lib/kubelet/pods/b8d8f634-6511-4e87-a35a-9cd55a4d8391/volumes"
Sep 17 15:55:38.025511 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025437    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"]
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025825    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025839    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025851    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="storage-initializer"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025857    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="storage-initializer"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025865    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kube-rbac-proxy"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025871    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kube-rbac-proxy"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025889    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="storage-initializer"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="storage-initializer"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025909    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kube-rbac-proxy"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kube-rbac-proxy"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025923    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="storage-initializer"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025928    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="storage-initializer"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025934    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025939    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kube-rbac-proxy"
Sep 17 15:55:38.025944 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.025953    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kube-rbac-proxy"
Sep 17 15:55:38.026674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.026002    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kube-rbac-proxy"
Sep 17 15:55:38.026674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.026012    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kube-rbac-proxy"
Sep 17 15:55:38.026674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.026020    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad066c00-8135-4aa1-b331-b2e72df331f1" containerName="kserve-container"
Sep 17 15:55:38.026674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.026030    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kserve-container"
Sep 17 15:55:38.026674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.026038    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10538fc5-d4ab-44e8-85cd-51b30f94123c" containerName="kube-rbac-proxy"
Sep 17 15:55:38.026674 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.026044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d8f634-6511-4e87-a35a-9cd55a4d8391" containerName="kserve-container"
Sep 17 15:55:38.028947 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.028927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.031515 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.031492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-3f9cc-kube-rbac-proxy-sar-config\""
Sep 17 15:55:38.031755 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.031736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-3f9cc-predictor-serving-cert\""
Sep 17 15:55:38.036752 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.036507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"]
Sep 17 15:55:38.036881 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.036838    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" containerID="cri-o://1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98" gracePeriod=30
Sep 17 15:55:38.036970 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.036846    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kube-rbac-proxy" containerID="cri-o://2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475" gracePeriod=30
Sep 17 15:55:38.042716 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.042694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"]
Sep 17 15:55:38.117043 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.117017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.117135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.117089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tsx\" (UniqueName: \"kubernetes.io/projected/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-api-access-b7tsx\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.117135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.117114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a06b4b9-846d-492a-bd73-cd0184cc1705-proxy-tls\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.213285 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.213261    2563 generic.go:356] "Generic (PLEG): container finished" podID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerID="2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475" exitCode=2
Sep 17 15:55:38.213369 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.213311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerDied","Data":"2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475"}
Sep 17 15:55:38.217555 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.217533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tsx\" (UniqueName: \"kubernetes.io/projected/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-api-access-b7tsx\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.217652 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.217572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a06b4b9-846d-492a-bd73-cd0184cc1705-proxy-tls\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.217652 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.217638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.218214 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.218192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.219775 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.219758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a06b4b9-846d-492a-bd73-cd0184cc1705-proxy-tls\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.230608 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.230589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tsx\" (UniqueName: \"kubernetes.io/projected/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-api-access-b7tsx\") pod \"message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.340530 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.340479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:38.620553 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.620473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"]
Sep 17 15:55:38.623566 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:55:38.623533    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a06b4b9_846d_492a_bd73_cd0184cc1705.slice/crio-b738c91e0bffb2a746b1ecb6ed6925e62746915e56e32e3e755b19015032e55a WatchSource:0}: Error finding container b738c91e0bffb2a746b1ecb6ed6925e62746915e56e32e3e755b19015032e55a: Status 404 returned error can't find the container with id b738c91e0bffb2a746b1ecb6ed6925e62746915e56e32e3e755b19015032e55a
Sep 17 15:55:38.625566 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:38.625546    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:55:39.219482 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:39.219443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" event={"ID":"8a06b4b9-846d-492a-bd73-cd0184cc1705","Type":"ContainerStarted","Data":"b738c91e0bffb2a746b1ecb6ed6925e62746915e56e32e3e755b19015032e55a"}
Sep 17 15:55:40.225908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:40.225823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" event={"ID":"8a06b4b9-846d-492a-bd73-cd0184cc1705","Type":"ContainerStarted","Data":"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"}
Sep 17 15:55:40.225908 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:40.225891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" event={"ID":"8a06b4b9-846d-492a-bd73-cd0184cc1705","Type":"ContainerStarted","Data":"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"}
Sep 17 15:55:40.226312 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:40.225925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:40.249142 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:40.249096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" podStartSLOduration=1.974240743 podStartE2EDuration="3.249080515s" totalImagesPullingTime="1.274839772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:40.24766479 +0000 UTC m=+993.209964347" watchObservedRunningTime="2026-09-17 15:55:40.249080515 +0000 UTC m=+993.211380063"
Sep 17 15:55:40.734832 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:40.734790    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 15:55:40.739413 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:40.739387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 15:55:41.230130 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.230098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:41.231661 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.231638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:55:41.380561 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.380535    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:55:41.443879 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.443822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsmxk\" (UniqueName: \"kubernetes.io/projected/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-api-access-dsmxk\") pod \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") "
Sep 17 15:55:41.444004 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.443883    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kserve-provision-location\") pod \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") "
Sep 17 15:55:41.444004 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.443946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-cabundle-cert\") pod \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") "
Sep 17 15:55:41.444004 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.443964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-proxy-tls\") pod \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") "
Sep 17 15:55:41.444004 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.443980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-rbac-proxy-sar-config\") pod \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\" (UID: \"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0\") "
Sep 17 15:55:41.444300 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.444278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" (UID: "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:41.444375 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.444319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" (UID: "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:41.444433 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.444409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" (UID: "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:41.445892 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.445872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" (UID: "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:41.446008 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.445991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-api-access-dsmxk" (OuterVolumeSpecName: "kube-api-access-dsmxk") pod "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" (UID: "aad0347c-d42b-4509-a0ec-4e9b2dc34ca0"). InnerVolumeSpecName "kube-api-access-dsmxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:41.545203 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.545186    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.545289 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.545205    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.545289 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.545214    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.545289 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.545223    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmxk\" (UniqueName: \"kubernetes.io/projected/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kube-api-access-dsmxk\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.545289 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:41.545232    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:55:42.236510 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.236477    2563 generic.go:356] "Generic (PLEG): container finished" podID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerID="1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98" exitCode=0
Sep 17 15:55:42.236851 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.236522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerDied","Data":"1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98"}
Sep 17 15:55:42.236851 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.236601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm" event={"ID":"aad0347c-d42b-4509-a0ec-4e9b2dc34ca0","Type":"ContainerDied","Data":"1638a9e771bf4dfb8fa6668c3bec7b7c4fe8f0783791b9ad4a6fa42b94a4e427"}
Sep 17 15:55:42.236851 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.236537    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"
Sep 17 15:55:42.236851 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.236632    2563 scope.go:117] "RemoveContainer" containerID="2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475"
Sep 17 15:55:42.245721 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.245697    2563 scope.go:117] "RemoveContainer" containerID="1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98"
Sep 17 15:55:42.253846 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.253676    2563 scope.go:117] "RemoveContainer" containerID="4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b"
Sep 17 15:55:42.260930 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.260903    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"]
Sep 17 15:55:42.262285 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.262263    2563 scope.go:117] "RemoveContainer" containerID="2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475"
Sep 17 15:55:42.262570 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:55:42.262547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475\": container with ID starting with 2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475 not found: ID does not exist" containerID="2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475"
Sep 17 15:55:42.262672 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.262576    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475"} err="failed to get container status \"2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475\": rpc error: code = NotFound desc = could not find container \"2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475\": container with ID starting with 2b699c61eb02a2bb0a71e9af17f700458ad23fbd1995e0297107190455395475 not found: ID does not exist"
Sep 17 15:55:42.262672 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.262594    2563 scope.go:117] "RemoveContainer" containerID="1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98"
Sep 17 15:55:42.262896 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:55:42.262880    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98\": container with ID starting with 1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98 not found: ID does not exist" containerID="1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98"
Sep 17 15:55:42.262939 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.262898    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98"} err="failed to get container status \"1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98\": rpc error: code = NotFound desc = could not find container \"1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98\": container with ID starting with 1d9393cba230bedf95ad4ad6196b0f8b88845b3e5995cae6dc542728c0531d98 not found: ID does not exist"
Sep 17 15:55:42.262939 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.262910    2563 scope.go:117] "RemoveContainer" containerID="4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b"
Sep 17 15:55:42.263151 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:55:42.263135    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b\": container with ID starting with 4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b not found: ID does not exist" containerID="4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b"
Sep 17 15:55:42.263190 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.263155    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b"} err="failed to get container status \"4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b\": rpc error: code = NotFound desc = could not find container \"4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b\": container with ID starting with 4768dc0a1f73abbf8c4a27f9871a80ab4b531b61daa7a15e3666aec357f2ad0b not found: ID does not exist"
Sep 17 15:55:42.266837 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:42.266813    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-883e7-predictor-5d96594d85-tbgpm"]
Sep 17 15:55:43.534574 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:43.534537    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" path="/var/lib/kubelet/pods/aad0347c-d42b-4509-a0ec-4e9b2dc34ca0/volumes"
Sep 17 15:55:48.247715 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:55:48.247671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:56:53.116260 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116206    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"]
Sep 17 15:56:53.116657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116554    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container"
Sep 17 15:56:53.116657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116564    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container"
Sep 17 15:56:53.116657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116583    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kube-rbac-proxy"
Sep 17 15:56:53.116657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116588    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kube-rbac-proxy"
Sep 17 15:56:53.116657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116600    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="storage-initializer"
Sep 17 15:56:53.116657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="storage-initializer"
Sep 17 15:56:53.116853 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116662    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kserve-container"
Sep 17 15:56:53.116853 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.116670    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad0347c-d42b-4509-a0ec-4e9b2dc34ca0" containerName="kube-rbac-proxy"
Sep 17 15:56:53.119743 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.119726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.122896 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.122871    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-e083cc-kube-rbac-proxy-sar-config\""
Sep 17 15:56:53.123272 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.123255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-e083cc-predictor-serving-cert\""
Sep 17 15:56:53.123933 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.123914    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-e083cc-dockercfg-n45t2\""
Sep 17 15:56:53.124036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.123922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-e083cc\""
Sep 17 15:56:53.124036 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.124002    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:56:53.132333 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.132312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"]
Sep 17 15:56:53.167456 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.167436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21870a1-b35a-406d-9bee-1d2820f06cbf-proxy-tls\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.167550 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.167461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-cabundle-cert\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.167550 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.167488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21870a1-b35a-406d-9bee-1d2820f06cbf-kserve-provision-location\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.167632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.167586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.167632 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.167624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9qwr\" (UniqueName: \"kubernetes.io/projected/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-api-access-m9qwr\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.268330 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.268306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9qwr\" (UniqueName: \"kubernetes.io/projected/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-api-access-m9qwr\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.268415 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.268357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21870a1-b35a-406d-9bee-1d2820f06cbf-proxy-tls\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.268415 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.268382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-cabundle-cert\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.268415 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.268411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21870a1-b35a-406d-9bee-1d2820f06cbf-kserve-provision-location\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.268523 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.268492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.268834 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.268806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21870a1-b35a-406d-9bee-1d2820f06cbf-kserve-provision-location\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.269135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.269114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.269135 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.269128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-cabundle-cert\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.270764 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.270748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21870a1-b35a-406d-9bee-1d2820f06cbf-proxy-tls\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.278291 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.278266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9qwr\" (UniqueName: \"kubernetes.io/projected/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-api-access-m9qwr\") pod \"isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") " pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.429543 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.429515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:56:53.706561 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:53.706539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"]
Sep 17 15:56:53.709343 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:56:53.709316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21870a1_b35a_406d_9bee_1d2820f06cbf.slice/crio-6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302 WatchSource:0}: Error finding container 6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302: Status 404 returned error can't find the container with id 6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302
Sep 17 15:56:54.558664 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:54.558623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" event={"ID":"f21870a1-b35a-406d-9bee-1d2820f06cbf","Type":"ContainerStarted","Data":"79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3"}
Sep 17 15:56:54.559030 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:54.558734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" event={"ID":"f21870a1-b35a-406d-9bee-1d2820f06cbf","Type":"ContainerStarted","Data":"6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302"}
Sep 17 15:56:59.583420 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:59.583384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/0.log"
Sep 17 15:56:59.583890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:59.583426    2563 generic.go:356] "Generic (PLEG): container finished" podID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerID="79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3" exitCode=1
Sep 17 15:56:59.583890 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:56:59.583512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" event={"ID":"f21870a1-b35a-406d-9bee-1d2820f06cbf","Type":"ContainerDied","Data":"79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3"}
Sep 17 15:57:00.590087 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:00.590016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/0.log"
Sep 17 15:57:00.590480 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:00.590121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" event={"ID":"f21870a1-b35a-406d-9bee-1d2820f06cbf","Type":"ContainerStarted","Data":"0f567dfca842b18ce8a2c3db393f4212b58ef077ec5b34e4c929b41b4a61512c"}
Sep 17 15:57:03.604463 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.604436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/1.log"
Sep 17 15:57:03.604867 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.604851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/0.log"
Sep 17 15:57:03.604912 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.604885    2563 generic.go:356] "Generic (PLEG): container finished" podID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerID="0f567dfca842b18ce8a2c3db393f4212b58ef077ec5b34e4c929b41b4a61512c" exitCode=1
Sep 17 15:57:03.604948 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.604913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" event={"ID":"f21870a1-b35a-406d-9bee-1d2820f06cbf","Type":"ContainerDied","Data":"0f567dfca842b18ce8a2c3db393f4212b58ef077ec5b34e4c929b41b4a61512c"}
Sep 17 15:57:03.604984 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.604967    2563 scope.go:117] "RemoveContainer" containerID="79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3"
Sep 17 15:57:03.605316 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.605292    2563 scope.go:117] "RemoveContainer" containerID="79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3"
Sep 17 15:57:03.616069 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:57:03.616042    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_kserve-ci-e2e-test(f21870a1-b35a-406d-9bee-1d2820f06cbf)\"" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf"
Sep 17 15:57:03.616375 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:57:03.616345    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_kserve-ci-e2e-test_f21870a1-b35a-406d-9bee-1d2820f06cbf_0 in pod sandbox 6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302 from index: no such id: '79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3'" containerID="79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3"
Sep 17 15:57:03.616448 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:03.616384    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_kserve-ci-e2e-test_f21870a1-b35a-406d-9bee-1d2820f06cbf_0 in pod sandbox 6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302 from index: no such id: '79fc4c727797c72257015233375695c059b1ebc81925bb841d5b59395e10e8c3'"
Sep 17 15:57:04.610541 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:04.610514    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/1.log"
Sep 17 15:57:11.152051 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.152012    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"]
Sep 17 15:57:11.271680 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.271661    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/1.log"
Sep 17 15:57:11.274043 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.274029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:57:11.309126 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-cabundle-cert\") pod \"f21870a1-b35a-406d-9bee-1d2820f06cbf\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") "
Sep 17 15:57:11.309209 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21870a1-b35a-406d-9bee-1d2820f06cbf-proxy-tls\") pod \"f21870a1-b35a-406d-9bee-1d2820f06cbf\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") "
Sep 17 15:57:11.309209 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9qwr\" (UniqueName: \"kubernetes.io/projected/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-api-access-m9qwr\") pod \"f21870a1-b35a-406d-9bee-1d2820f06cbf\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") "
Sep 17 15:57:11.309307 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-rbac-proxy-sar-config\") pod \"f21870a1-b35a-406d-9bee-1d2820f06cbf\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") "
Sep 17 15:57:11.309343 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21870a1-b35a-406d-9bee-1d2820f06cbf-kserve-provision-location\") pod \"f21870a1-b35a-406d-9bee-1d2820f06cbf\" (UID: \"f21870a1-b35a-406d-9bee-1d2820f06cbf\") "
Sep 17 15:57:11.309592 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f21870a1-b35a-406d-9bee-1d2820f06cbf" (UID: "f21870a1-b35a-406d-9bee-1d2820f06cbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:11.309663 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21870a1-b35a-406d-9bee-1d2820f06cbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f21870a1-b35a-406d-9bee-1d2820f06cbf" (UID: "f21870a1-b35a-406d-9bee-1d2820f06cbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:11.309663 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.309653    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f21870a1-b35a-406d-9bee-1d2820f06cbf" (UID: "f21870a1-b35a-406d-9bee-1d2820f06cbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:11.311204 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.311183    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-api-access-m9qwr" (OuterVolumeSpecName: "kube-api-access-m9qwr") pod "f21870a1-b35a-406d-9bee-1d2820f06cbf" (UID: "f21870a1-b35a-406d-9bee-1d2820f06cbf"). InnerVolumeSpecName "kube-api-access-m9qwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:11.311204 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.311196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21870a1-b35a-406d-9bee-1d2820f06cbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f21870a1-b35a-406d-9bee-1d2820f06cbf" (UID: "f21870a1-b35a-406d-9bee-1d2820f06cbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:11.410682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.410632    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f21870a1-b35a-406d-9bee-1d2820f06cbf-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:11.410682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.410652    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:11.410682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.410662    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f21870a1-b35a-406d-9bee-1d2820f06cbf-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:11.410682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.410670    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9qwr\" (UniqueName: \"kubernetes.io/projected/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-api-access-m9qwr\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:11.410682 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.410678    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f21870a1-b35a-406d-9bee-1d2820f06cbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:11.642123 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.642098    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg_f21870a1-b35a-406d-9bee-1d2820f06cbf/storage-initializer/1.log"
Sep 17 15:57:11.642227 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.642198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"
Sep 17 15:57:11.642308 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.642235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg" event={"ID":"f21870a1-b35a-406d-9bee-1d2820f06cbf","Type":"ContainerDied","Data":"6b09e64fa13ac379efd49e436ce661fcdc2872bcf87ab959f9cf3de84b428302"}
Sep 17 15:57:11.642347 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.642335    2563 scope.go:117] "RemoveContainer" containerID="0f567dfca842b18ce8a2c3db393f4212b58ef077ec5b34e4c929b41b4a61512c"
Sep 17 15:57:11.685034 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.684972    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"]
Sep 17 15:57:11.692939 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:11.692919    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e083cc-predictor-74dd6cc75f-vdmvg"]
Sep 17 15:57:13.535343 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:13.535299    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" path="/var/lib/kubelet/pods/f21870a1-b35a-406d-9bee-1d2820f06cbf/volumes"
Sep 17 15:57:33.105373 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.105333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d_8a06b4b9-846d-492a-bd73-cd0184cc1705/kserve-container/0.log"
Sep 17 15:57:33.353768 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.353733    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"]
Sep 17 15:57:33.353973 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.353946    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kserve-container" containerID="cri-o://9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf" gracePeriod=30
Sep 17 15:57:33.354026 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.353981    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kube-rbac-proxy" containerID="cri-o://9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608" gracePeriod=30
Sep 17 15:57:33.537356 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537323    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"]
Sep 17 15:57:33.537774 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537754    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerName="storage-initializer"
Sep 17 15:57:33.537774 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537773    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerName="storage-initializer"
Sep 17 15:57:33.537980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537856    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerName="storage-initializer"
Sep 17 15:57:33.537980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerName="storage-initializer"
Sep 17 15:57:33.537980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537930    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerName="storage-initializer"
Sep 17 15:57:33.537980 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.537939    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21870a1-b35a-406d-9bee-1d2820f06cbf" containerName="storage-initializer"
Sep 17 15:57:33.542705 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.542681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.546657 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.546633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5aee0-kube-rbac-proxy-sar-config\""
Sep 17 15:57:33.546778 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.546661    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:57:33.547048 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.547029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:57:33.549071 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.549053    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5aee0-predictor-serving-cert\""
Sep 17 15:57:33.614847 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.614803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:57:33.642723 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.642700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"]
Sep 17 15:57:33.675448 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7tsx\" (UniqueName: \"kubernetes.io/projected/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-api-access-b7tsx\") pod \"8a06b4b9-846d-492a-bd73-cd0184cc1705\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") "
Sep 17 15:57:33.675549 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-rbac-proxy-sar-config\") pod \"8a06b4b9-846d-492a-bd73-cd0184cc1705\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") "
Sep 17 15:57:33.675549 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a06b4b9-846d-492a-bd73-cd0184cc1705-proxy-tls\") pod \"8a06b4b9-846d-492a-bd73-cd0184cc1705\" (UID: \"8a06b4b9-846d-492a-bd73-cd0184cc1705\") "
Sep 17 15:57:33.675669 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqbs\" (UniqueName: \"kubernetes.io/projected/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-api-access-8tqbs\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.675669 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.675669 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.675669 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.675867 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.675867 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a06b4b9-846d-492a-bd73-cd0184cc1705" (UID: "8a06b4b9-846d-492a-bd73-cd0184cc1705"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:33.675932 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.675866    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:33.677429 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.677410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-api-access-b7tsx" (OuterVolumeSpecName: "kube-api-access-b7tsx") pod "8a06b4b9-846d-492a-bd73-cd0184cc1705" (UID: "8a06b4b9-846d-492a-bd73-cd0184cc1705"). InnerVolumeSpecName "kube-api-access-b7tsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:33.677496 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.677471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a06b4b9-846d-492a-bd73-cd0184cc1705-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a06b4b9-846d-492a-bd73-cd0184cc1705" (UID: "8a06b4b9-846d-492a-bd73-cd0184cc1705"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:33.744256 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744221    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerID="9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608" exitCode=2
Sep 17 15:57:33.744256 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744252    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerID="9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf" exitCode=2
Sep 17 15:57:33.744369 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"
Sep 17 15:57:33.744369 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" event={"ID":"8a06b4b9-846d-492a-bd73-cd0184cc1705","Type":"ContainerDied","Data":"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"}
Sep 17 15:57:33.744369 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" event={"ID":"8a06b4b9-846d-492a-bd73-cd0184cc1705","Type":"ContainerDied","Data":"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"}
Sep 17 15:57:33.744461 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d" event={"ID":"8a06b4b9-846d-492a-bd73-cd0184cc1705","Type":"ContainerDied","Data":"b738c91e0bffb2a746b1ecb6ed6925e62746915e56e32e3e755b19015032e55a"}
Sep 17 15:57:33.744461 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.744413    2563 scope.go:117] "RemoveContainer" containerID="9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"
Sep 17 15:57:33.752769 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.752752    2563 scope.go:117] "RemoveContainer" containerID="9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"
Sep 17 15:57:33.759951 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.759935    2563 scope.go:117] "RemoveContainer" containerID="9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"
Sep 17 15:57:33.760211 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:57:33.760195    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608\": container with ID starting with 9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608 not found: ID does not exist" containerID="9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"
Sep 17 15:57:33.760282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760218    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"} err="failed to get container status \"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608\": rpc error: code = NotFound desc = could not find container \"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608\": container with ID starting with 9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608 not found: ID does not exist"
Sep 17 15:57:33.760282 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760232    2563 scope.go:117] "RemoveContainer" containerID="9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"
Sep 17 15:57:33.760462 ip-10-0-143-66 kubenswrapper[2563]: E0917 15:57:33.760445    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf\": container with ID starting with 9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf not found: ID does not exist" containerID="9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"
Sep 17 15:57:33.760502 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"} err="failed to get container status \"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf\": rpc error: code = NotFound desc = could not find container \"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf\": container with ID starting with 9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf not found: ID does not exist"
Sep 17 15:57:33.760502 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760479    2563 scope.go:117] "RemoveContainer" containerID="9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"
Sep 17 15:57:33.760717 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760701    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608"} err="failed to get container status \"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608\": rpc error: code = NotFound desc = could not find container \"9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608\": container with ID starting with 9b02f23ca68b68ebe03cb689293eb2180bd606bc9e5580c74d293a20ce7d8608 not found: ID does not exist"
Sep 17 15:57:33.760717 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760716    2563 scope.go:117] "RemoveContainer" containerID="9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"
Sep 17 15:57:33.760934 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.760919    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf"} err="failed to get container status \"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf\": rpc error: code = NotFound desc = could not find container \"9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf\": container with ID starting with 9f2d0cc1a11afa2204011788eb08896fbaa789185e34e8102223b89d809a47cf not found: ID does not exist"
Sep 17 15:57:33.768099 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.768078    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"]
Sep 17 15:57:33.775707 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.775685    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f9cc-predictor-7f56659f5d-nct2d"]
Sep 17 15:57:33.776342 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.776393 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.776484 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqbs\" (UniqueName: \"kubernetes.io/projected/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-api-access-8tqbs\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.776536 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.776536 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.776623 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776573    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tsx\" (UniqueName: \"kubernetes.io/projected/8a06b4b9-846d-492a-bd73-cd0184cc1705-kube-api-access-b7tsx\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:33.776623 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776589    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a06b4b9-846d-492a-bd73-cd0184cc1705-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 15:57:33.776832 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.776814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.777040 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.777022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.777119 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.777101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.778398 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.778384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.786326 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.786306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqbs\" (UniqueName: \"kubernetes.io/projected/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-api-access-8tqbs\") pod \"isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:33.853440 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:33.853420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:34.135398 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:34.135368    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"]
Sep 17 15:57:34.138539 ip-10-0-143-66 kubenswrapper[2563]: W0917 15:57:34.138512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd0b8c9_7b80_41f0_b4c8_316a82c166b0.slice/crio-eaf5d756987e9bf2afbaa0dea6ec3cc2a40f0eb38b00a65505b1f6efe3eb2dc1 WatchSource:0}: Error finding container eaf5d756987e9bf2afbaa0dea6ec3cc2a40f0eb38b00a65505b1f6efe3eb2dc1: Status 404 returned error can't find the container with id eaf5d756987e9bf2afbaa0dea6ec3cc2a40f0eb38b00a65505b1f6efe3eb2dc1
Sep 17 15:57:34.752473 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:34.752441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerStarted","Data":"0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3"}
Sep 17 15:57:34.752631 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:34.752498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerStarted","Data":"eaf5d756987e9bf2afbaa0dea6ec3cc2a40f0eb38b00a65505b1f6efe3eb2dc1"}
Sep 17 15:57:35.535167 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:35.535129    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" path="/var/lib/kubelet/pods/8a06b4b9-846d-492a-bd73-cd0184cc1705/volumes"
Sep 17 15:57:38.773161 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:38.773120    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerID="0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3" exitCode=0
Sep 17 15:57:38.773681 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:38.773199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerDied","Data":"0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3"}
Sep 17 15:57:39.780033 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:39.780001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerStarted","Data":"8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f"}
Sep 17 15:57:39.780453 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:39.780059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerStarted","Data":"559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315"}
Sep 17 15:57:39.780453 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:39.780269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:39.806641 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:39.806592    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podStartSLOduration=2.99285016 podStartE2EDuration="6.806577426s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813727266s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:39.80433285 +0000 UTC m=+1112.766632414" watchObservedRunningTime="2026-09-17 15:57:39.806577426 +0000 UTC m=+1112.768876971"
Sep 17 15:57:40.785190 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:40.785146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:40.786544 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:40.786514    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:57:41.789578 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:41.789535    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:57:46.794162 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:46.794134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 15:57:46.794594 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:46.794563    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:57:56.794629 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:57:56.794582    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:58:06.795409 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:58:06.795365    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:58:16.795601 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:58:16.795550    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:58:26.800529 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:58:26.800480    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:58:36.794643 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:58:36.794550    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:58:46.794753 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:58:46.794716    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:58:56.794999 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:58:56.794937    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:59:00.529650 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:59:00.529608    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:59:10.530528 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:59:10.530481    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:59:20.529563 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:59:20.529520    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:59:30.529921 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:59:30.529872    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:59:40.530043 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:59:40.529994    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 15:59:50.529843 ip-10-0-143-66 kubenswrapper[2563]: I0917 15:59:50.529789    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 16:00:00.529674 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:00.529578    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 16:00:10.530197 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:10.530160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 16:00:13.575102 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:13.575065    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"]
Sep 17 16:00:13.575530 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:13.575382    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" containerID="cri-o://559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315" gracePeriod=30
Sep 17 16:00:13.575530 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:13.575457    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kube-rbac-proxy" containerID="cri-o://8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f" gracePeriod=30
Sep 17 16:00:14.457675 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:14.457638    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerID="8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f" exitCode=2
Sep 17 16:00:14.457854 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:14.457715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerDied","Data":"8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f"}
Sep 17 16:00:16.790061 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:16.790019    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 16:00:18.787649 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.787616    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"]
Sep 17 16:00:18.788013 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.787981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kube-rbac-proxy"
Sep 17 16:00:18.788013 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.787994    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kube-rbac-proxy"
Sep 17 16:00:18.788013 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.788011    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kserve-container"
Sep 17 16:00:18.788116 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.788017    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kserve-container"
Sep 17 16:00:18.788116 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.788079    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kserve-container"
Sep 17 16:00:18.788116 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.788092    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a06b4b9-846d-492a-bd73-cd0184cc1705" containerName="kube-rbac-proxy"
Sep 17 16:00:18.791320 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.791299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.794408 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.794385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:00:18.799687 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.799664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:00:18.802229 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.802204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.802354 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.802296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6317442-1772-4c50-8506-015c53d2f4b5-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.802354 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.802322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xss\" (UniqueName: \"kubernetes.io/projected/c6317442-1772-4c50-8506-015c53d2f4b5-kube-api-access-s6xss\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.802427 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.802359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.802427 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.802376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.810439 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.810412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"]
Sep 17 16:00:18.902977 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.902923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6317442-1772-4c50-8506-015c53d2f4b5-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.902977 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.902976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xss\" (UniqueName: \"kubernetes.io/projected/c6317442-1772-4c50-8506-015c53d2f4b5-kube-api-access-s6xss\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.903235 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.903010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.903235 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.903034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.903235 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.903085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.903235 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:00:18.903229    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 16:00:18.903477 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:00:18.903317    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls podName:c6317442-1772-4c50-8506-015c53d2f4b5 nodeName:}" failed. No retries permitted until 2026-09-17 16:00:19.403296272 +0000 UTC m=+1272.365595816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls") pod "isvc-canary-create-predictor-5674b8c456-q6bjq" (UID: "c6317442-1772-4c50-8506-015c53d2f4b5") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 16:00:18.903477 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.903441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6317442-1772-4c50-8506-015c53d2f4b5-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.903794 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.903775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.903841 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.903827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:18.933463 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:18.933433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xss\" (UniqueName: \"kubernetes.io/projected/c6317442-1772-4c50-8506-015c53d2f4b5-kube-api-access-s6xss\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:19.407981 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:19.407938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:19.410443 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:19.410415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-q6bjq\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:19.701659 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:19.701551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:19.982632 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:19.982544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"]
Sep 17 16:00:19.985812 ip-10-0-143-66 kubenswrapper[2563]: W0917 16:00:19.985780    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6317442_1772_4c50_8506_015c53d2f4b5.slice/crio-c0d5ae80b67c0bf7cb16b1d4b5be246f2f574ecf46433c8a7c2eb2c496d00cd0 WatchSource:0}: Error finding container c0d5ae80b67c0bf7cb16b1d4b5be246f2f574ecf46433c8a7c2eb2c496d00cd0: Status 404 returned error can't find the container with id c0d5ae80b67c0bf7cb16b1d4b5be246f2f574ecf46433c8a7c2eb2c496d00cd0
Sep 17 16:00:20.488372 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:20.488331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerStarted","Data":"73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f"}
Sep 17 16:00:20.488577 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:20.488409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerStarted","Data":"c0d5ae80b67c0bf7cb16b1d4b5be246f2f574ecf46433c8a7c2eb2c496d00cd0"}
Sep 17 16:00:20.529785 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:20.529742    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 16:00:21.790601 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:21.790560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 16:00:24.177485 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.177458    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 16:00:24.254044 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.253966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tqbs\" (UniqueName: \"kubernetes.io/projected/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-api-access-8tqbs\") pod \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") "
Sep 17 16:00:24.254044 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-cabundle-cert\") pod \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") "
Sep 17 16:00:24.254044 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-proxy-tls\") pod \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") "
Sep 17 16:00:24.254292 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kserve-provision-location\") pod \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") "
Sep 17 16:00:24.254292 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-rbac-proxy-sar-config\") pod \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\" (UID: \"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0\") "
Sep 17 16:00:24.254493 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" (UID: "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:24.254579 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254476    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" (UID: "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:24.254625 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.254566    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" (UID: "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:24.256088 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.256070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-api-access-8tqbs" (OuterVolumeSpecName: "kube-api-access-8tqbs") pod "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" (UID: "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0"). InnerVolumeSpecName "kube-api-access-8tqbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:00:24.256155 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.256142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" (UID: "fdd0b8c9-7b80-41f0-b4c8-316a82c166b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:00:24.354976 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.354932    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:00:24.354976 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.354969    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqbs\" (UniqueName: \"kubernetes.io/projected/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kube-api-access-8tqbs\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:00:24.354976 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.354980    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:00:24.355210 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.354989    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:00:24.355210 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.354997    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:00:24.506773 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.506680    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6317442-1772-4c50-8506-015c53d2f4b5" containerID="73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f" exitCode=0
Sep 17 16:00:24.506773 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.506753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerDied","Data":"73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f"}
Sep 17 16:00:24.508835 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.508810    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerID="559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315" exitCode=0
Sep 17 16:00:24.508948 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.508854    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"
Sep 17 16:00:24.508948 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.508891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerDied","Data":"559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315"}
Sep 17 16:00:24.509068 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.508964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl" event={"ID":"fdd0b8c9-7b80-41f0-b4c8-316a82c166b0","Type":"ContainerDied","Data":"eaf5d756987e9bf2afbaa0dea6ec3cc2a40f0eb38b00a65505b1f6efe3eb2dc1"}
Sep 17 16:00:24.509068 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.509006    2563 scope.go:117] "RemoveContainer" containerID="8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f"
Sep 17 16:00:24.521045 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.521025    2563 scope.go:117] "RemoveContainer" containerID="559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315"
Sep 17 16:00:24.531963 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.531804    2563 scope.go:117] "RemoveContainer" containerID="0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3"
Sep 17 16:00:24.545907 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.545880    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"]
Sep 17 16:00:24.547379 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.547355    2563 scope.go:117] "RemoveContainer" containerID="8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f"
Sep 17 16:00:24.547690 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:00:24.547671    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f\": container with ID starting with 8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f not found: ID does not exist" containerID="8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f"
Sep 17 16:00:24.547747 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.547700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f"} err="failed to get container status \"8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f\": rpc error: code = NotFound desc = could not find container \"8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f\": container with ID starting with 8851c76466af150c57aa56207cacbb871ce6aed2308febd9c002da52f7eb578f not found: ID does not exist"
Sep 17 16:00:24.547747 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.547718    2563 scope.go:117] "RemoveContainer" containerID="559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315"
Sep 17 16:00:24.547995 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:00:24.547977    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315\": container with ID starting with 559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315 not found: ID does not exist" containerID="559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315"
Sep 17 16:00:24.548070 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.548004    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315"} err="failed to get container status \"559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315\": rpc error: code = NotFound desc = could not find container \"559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315\": container with ID starting with 559260f275b80a2b6aaeaa7df76d485042492115008c6b32f3f41b47659af315 not found: ID does not exist"
Sep 17 16:00:24.548070 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.548026    2563 scope.go:117] "RemoveContainer" containerID="0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3"
Sep 17 16:00:24.548285 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:00:24.548268    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3\": container with ID starting with 0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3 not found: ID does not exist" containerID="0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3"
Sep 17 16:00:24.548345 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.548291    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3"} err="failed to get container status \"0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3\": rpc error: code = NotFound desc = could not find container \"0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3\": container with ID starting with 0f804a7d94cdf3de14a4f671e0e1ab8f5cbb24c0396cc7c4ba27c1025bfec3f3 not found: ID does not exist"
Sep 17 16:00:24.552592 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:24.552571    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5aee0-predictor-c4c8c647f-f67pl"]
Sep 17 16:00:25.515971 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.515937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerStarted","Data":"ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91"}
Sep 17 16:00:25.516475 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.516007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerStarted","Data":"7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c"}
Sep 17 16:00:25.516475 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.516278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:25.516475 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.516342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:25.517960 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.517932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:00:25.534741 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.534705    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" path="/var/lib/kubelet/pods/fdd0b8c9-7b80-41f0-b4c8-316a82c166b0/volumes"
Sep 17 16:00:25.554211 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:25.554138    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podStartSLOduration=3.497406782 podStartE2EDuration="7.554120496s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.056713714s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:25.552747893 +0000 UTC m=+1278.515047464" watchObservedRunningTime="2026-09-17 16:00:25.554120496 +0000 UTC m=+1278.516420041"
Sep 17 16:00:26.521512 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:26.521474    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:00:31.526146 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:31.526118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:00:31.526765 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:31.526734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:00:41.526602 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:41.526557    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:00:51.527085 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:00:51.527040    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:01:01.527061 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:01.527018    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:01:11.526997 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:11.526959    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:01:21.526883 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:21.526841    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:01:31.527466 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:31.527378    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:01:36.530489 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:36.530451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:01:48.837541 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:48.837510    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"]
Sep 17 16:01:48.838017 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:48.837731    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container" containerID="cri-o://7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c" gracePeriod=30
Sep 17 16:01:48.838017 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:48.837770    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kube-rbac-proxy" containerID="cri-o://ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91" gracePeriod=30
Sep 17 16:01:49.877597 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:49.877563    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6317442-1772-4c50-8506-015c53d2f4b5" containerID="ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91" exitCode=2
Sep 17 16:01:49.877939 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:49.877636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerDied","Data":"ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91"}
Sep 17 16:01:51.522600 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:51.522560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 16:01:52.373200 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.373177    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:01:52.422042 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.421980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6317442-1772-4c50-8506-015c53d2f4b5-kserve-provision-location\") pod \"c6317442-1772-4c50-8506-015c53d2f4b5\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") "
Sep 17 16:01:52.422156 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422054    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls\") pod \"c6317442-1772-4c50-8506-015c53d2f4b5\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") "
Sep 17 16:01:52.422156 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-kube-rbac-proxy-sar-config\") pod \"c6317442-1772-4c50-8506-015c53d2f4b5\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") "
Sep 17 16:01:52.422156 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6xss\" (UniqueName: \"kubernetes.io/projected/c6317442-1772-4c50-8506-015c53d2f4b5-kube-api-access-s6xss\") pod \"c6317442-1772-4c50-8506-015c53d2f4b5\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") "
Sep 17 16:01:52.422156 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-cabundle-cert\") pod \"c6317442-1772-4c50-8506-015c53d2f4b5\" (UID: \"c6317442-1772-4c50-8506-015c53d2f4b5\") "
Sep 17 16:01:52.422482 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6317442-1772-4c50-8506-015c53d2f4b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6317442-1772-4c50-8506-015c53d2f4b5" (UID: "c6317442-1772-4c50-8506-015c53d2f4b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:52.422482 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6317442-1772-4c50-8506-015c53d2f4b5" (UID: "c6317442-1772-4c50-8506-015c53d2f4b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:52.422482 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6317442-1772-4c50-8506-015c53d2f4b5" (UID: "c6317442-1772-4c50-8506-015c53d2f4b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:52.422482 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.422478    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6317442-1772-4c50-8506-015c53d2f4b5-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:01:52.424043 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.424019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6317442-1772-4c50-8506-015c53d2f4b5" (UID: "c6317442-1772-4c50-8506-015c53d2f4b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:52.424172 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.424157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6317442-1772-4c50-8506-015c53d2f4b5-kube-api-access-s6xss" (OuterVolumeSpecName: "kube-api-access-s6xss") pod "c6317442-1772-4c50-8506-015c53d2f4b5" (UID: "c6317442-1772-4c50-8506-015c53d2f4b5"). InnerVolumeSpecName "kube-api-access-s6xss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:52.523642 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.523620    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6317442-1772-4c50-8506-015c53d2f4b5-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:01:52.523642 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.523640    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:01:52.523937 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.523649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xss\" (UniqueName: \"kubernetes.io/projected/c6317442-1772-4c50-8506-015c53d2f4b5-kube-api-access-s6xss\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:01:52.523937 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.523658    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6317442-1772-4c50-8506-015c53d2f4b5-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:01:52.891055 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.891032    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6317442-1772-4c50-8506-015c53d2f4b5" containerID="7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c" exitCode=0
Sep 17 16:01:52.891155 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.891063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerDied","Data":"7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c"}
Sep 17 16:01:52.891155 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.891090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"
Sep 17 16:01:52.891155 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.891106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq" event={"ID":"c6317442-1772-4c50-8506-015c53d2f4b5","Type":"ContainerDied","Data":"c0d5ae80b67c0bf7cb16b1d4b5be246f2f574ecf46433c8a7c2eb2c496d00cd0"}
Sep 17 16:01:52.891155 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.891132    2563 scope.go:117] "RemoveContainer" containerID="ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91"
Sep 17 16:01:52.899767 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.899749    2563 scope.go:117] "RemoveContainer" containerID="7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c"
Sep 17 16:01:52.906558 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.906538    2563 scope.go:117] "RemoveContainer" containerID="73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f"
Sep 17 16:01:52.913517 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.913502    2563 scope.go:117] "RemoveContainer" containerID="ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91"
Sep 17 16:01:52.913775 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:01:52.913758    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91\": container with ID starting with ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91 not found: ID does not exist" containerID="ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91"
Sep 17 16:01:52.913835 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.913782    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91"} err="failed to get container status \"ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91\": rpc error: code = NotFound desc = could not find container \"ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91\": container with ID starting with ecdf49a7b27a8cb37849a6efe06e000a97602290185333bcec6a926149587d91 not found: ID does not exist"
Sep 17 16:01:52.913835 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.913798    2563 scope.go:117] "RemoveContainer" containerID="7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c"
Sep 17 16:01:52.914034 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:01:52.914017    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c\": container with ID starting with 7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c not found: ID does not exist" containerID="7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c"
Sep 17 16:01:52.914076 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.914041    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c"} err="failed to get container status \"7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c\": rpc error: code = NotFound desc = could not find container \"7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c\": container with ID starting with 7b671cdc2f6a60c582cba661b533887dd197519004a34a41d57694945d5fb62c not found: ID does not exist"
Sep 17 16:01:52.914076 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.914056    2563 scope.go:117] "RemoveContainer" containerID="73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f"
Sep 17 16:01:52.914321 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:01:52.914306    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f\": container with ID starting with 73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f not found: ID does not exist" containerID="73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f"
Sep 17 16:01:52.914387 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.914325    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f"} err="failed to get container status \"73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f\": rpc error: code = NotFound desc = could not find container \"73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f\": container with ID starting with 73220a503ac56953abc9365ae2f38908734023ace39828706368841c0491a85f not found: ID does not exist"
Sep 17 16:01:52.924347 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.924328    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"]
Sep 17 16:01:52.937813 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:52.937793    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-q6bjq"]
Sep 17 16:01:53.534183 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:53.534158    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" path="/var/lib/kubelet/pods/c6317442-1772-4c50-8506-015c53d2f4b5/volumes"
Sep 17 16:01:54.065799 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.065772    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"]
Sep 17 16:01:54.066141 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066129    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="storage-initializer"
Sep 17 16:01:54.066187 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066143    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="storage-initializer"
Sep 17 16:01:54.066187 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066156    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container"
Sep 17 16:01:54.066187 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066162    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container"
Sep 17 16:01:54.066187 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066177    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kube-rbac-proxy"
Sep 17 16:01:54.066187 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066185    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kube-rbac-proxy"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066201    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066206    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066213    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="storage-initializer"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066219    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="storage-initializer"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kube-rbac-proxy"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066234    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kube-rbac-proxy"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kserve-container"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066308    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kube-rbac-proxy"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066314    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6317442-1772-4c50-8506-015c53d2f4b5" containerName="kserve-container"
Sep 17 16:01:54.066378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.066320    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd0b8c9-7b80-41f0-b4c8-316a82c166b0" containerName="kube-rbac-proxy"
Sep 17 16:01:54.071002 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.070985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.073926 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.073908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:01:54.074309 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.074292    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:01:54.074378 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.074341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:01:54.075014 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.074987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:01:54.075124 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.075064    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:01:54.075185 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.075144    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:01:54.075237 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.075183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 16:01:54.081918 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.081899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"]
Sep 17 16:01:54.135970 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.135947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9srxc\" (UniqueName: \"kubernetes.io/projected/d54555c1-98a5-4186-9876-8c522df04070-kube-api-access-9srxc\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.136055 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.135987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d54555c1-98a5-4186-9876-8c522df04070-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.136055 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.136010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.136131 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.136080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.136131 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.136107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54555c1-98a5-4186-9876-8c522df04070-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237071 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9srxc\" (UniqueName: \"kubernetes.io/projected/d54555c1-98a5-4186-9876-8c522df04070-kube-api-access-9srxc\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237173 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d54555c1-98a5-4186-9876-8c522df04070-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237173 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237173 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237373 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54555c1-98a5-4186-9876-8c522df04070-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237572 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d54555c1-98a5-4186-9876-8c522df04070-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.237939 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.238027 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.237916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.239550 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.239531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54555c1-98a5-4186-9876-8c522df04070-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.251100 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.251079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9srxc\" (UniqueName: \"kubernetes.io/projected/d54555c1-98a5-4186-9876-8c522df04070-kube-api-access-9srxc\") pod \"isvc-canary-promote-predictor-8499c568cb-qp4kt\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.382112 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.382090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:54.656799 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.656777    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"]
Sep 17 16:01:54.658385 ip-10-0-143-66 kubenswrapper[2563]: W0917 16:01:54.658361    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54555c1_98a5_4186_9876_8c522df04070.slice/crio-7bcd7645790a34454f18e5190b395b045faf032f6119659295b643d9e22846c0 WatchSource:0}: Error finding container 7bcd7645790a34454f18e5190b395b045faf032f6119659295b643d9e22846c0: Status 404 returned error can't find the container with id 7bcd7645790a34454f18e5190b395b045faf032f6119659295b643d9e22846c0
Sep 17 16:01:54.902427 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.902390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerStarted","Data":"344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5"}
Sep 17 16:01:54.902610 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:54.902448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerStarted","Data":"7bcd7645790a34454f18e5190b395b045faf032f6119659295b643d9e22846c0"}
Sep 17 16:01:58.918410 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:58.918347    2563 generic.go:356] "Generic (PLEG): container finished" podID="d54555c1-98a5-4186-9876-8c522df04070" containerID="344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5" exitCode=0
Sep 17 16:01:58.918410 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:58.918383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerDied","Data":"344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5"}
Sep 17 16:01:59.925326 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:59.925297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerStarted","Data":"d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54"}
Sep 17 16:01:59.925747 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:59.925364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerStarted","Data":"b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9"}
Sep 17 16:01:59.925747 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:59.925572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:01:59.947185 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:01:59.947143    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podStartSLOduration=2.03485062 podStartE2EDuration="5.94713013s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.91227951s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:59.945930172 +0000 UTC m=+1372.908229727" watchObservedRunningTime="2026-09-17 16:01:59.94713013 +0000 UTC m=+1372.909429675"
Sep 17 16:02:00.931092 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:00.931059    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:02:00.932319 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:00.932281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:01.934771 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:01.934732    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:06.939493 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:06.939466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:02:06.940042 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:06.940013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:16.940049 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:16.940006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:26.940200 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:26.940154    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:36.940069 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:36.940030    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:46.940624 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:46.940583    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:02:56.940204 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:02:56.940162    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:03:06.940912 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:06.940830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:03:14.065292 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.065237    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"]
Sep 17 16:03:14.065774 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.065575    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" containerID="cri-o://b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9" gracePeriod=30
Sep 17 16:03:14.065774 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.065605    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kube-rbac-proxy" containerID="cri-o://d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54" gracePeriod=30
Sep 17 16:03:14.124002 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.123971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"]
Sep 17 16:03:14.127381 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.127365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.131149 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.131128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:03:14.142563 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.142539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"]
Sep 17 16:03:14.216535 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.216511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.216637 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.216561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.216637 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.216620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.216739 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.216645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83eb7707-28fa-4f58-8416-c17fbf7a95f1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.216739 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.216670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm28v\" (UniqueName: \"kubernetes.io/projected/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-api-access-lm28v\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.239586 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.239563    2563 generic.go:356] "Generic (PLEG): container finished" podID="d54555c1-98a5-4186-9876-8c522df04070" containerID="d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54" exitCode=2
Sep 17 16:03:14.239676 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.239605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerDied","Data":"d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54"}
Sep 17 16:03:14.317450 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.317450 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83eb7707-28fa-4f58-8416-c17fbf7a95f1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.317450 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28v\" (UniqueName: \"kubernetes.io/projected/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-api-access-lm28v\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.317628 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.317628 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.317900 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.318015 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.317994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.318070 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.318053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.319585 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.319568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83eb7707-28fa-4f58-8416-c17fbf7a95f1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.329186 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.329164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm28v\" (UniqueName: \"kubernetes.io/projected/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-api-access-lm28v\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-r45v6\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.438348 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.438324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:14.711717 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:14.711694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"]
Sep 17 16:03:14.713996 ip-10-0-143-66 kubenswrapper[2563]: W0917 16:03:14.713966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eb7707_28fa_4f58_8416_c17fbf7a95f1.slice/crio-38014b9b49efcaeb2dc96ab1a21642857d295b1717eaa9ee7d3c3d44e7430e12 WatchSource:0}: Error finding container 38014b9b49efcaeb2dc96ab1a21642857d295b1717eaa9ee7d3c3d44e7430e12: Status 404 returned error can't find the container with id 38014b9b49efcaeb2dc96ab1a21642857d295b1717eaa9ee7d3c3d44e7430e12
Sep 17 16:03:15.247219 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:15.247185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerStarted","Data":"5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f"}
Sep 17 16:03:15.247666 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:15.247270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerStarted","Data":"38014b9b49efcaeb2dc96ab1a21642857d295b1717eaa9ee7d3c3d44e7430e12"}
Sep 17 16:03:16.935971 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:16.935931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 16:03:16.940286 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:16.940236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:03:18.102554 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.102532    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:03:18.251398 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9srxc\" (UniqueName: \"kubernetes.io/projected/d54555c1-98a5-4186-9876-8c522df04070-kube-api-access-9srxc\") pod \"d54555c1-98a5-4186-9876-8c522df04070\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") "
Sep 17 16:03:18.251398 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d54555c1-98a5-4186-9876-8c522df04070-kserve-provision-location\") pod \"d54555c1-98a5-4186-9876-8c522df04070\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") "
Sep 17 16:03:18.251577 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-kube-rbac-proxy-sar-config\") pod \"d54555c1-98a5-4186-9876-8c522df04070\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") "
Sep 17 16:03:18.251577 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-cabundle-cert\") pod \"d54555c1-98a5-4186-9876-8c522df04070\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") "
Sep 17 16:03:18.251577 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251474    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54555c1-98a5-4186-9876-8c522df04070-proxy-tls\") pod \"d54555c1-98a5-4186-9876-8c522df04070\" (UID: \"d54555c1-98a5-4186-9876-8c522df04070\") "
Sep 17 16:03:18.251728 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54555c1-98a5-4186-9876-8c522df04070-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d54555c1-98a5-4186-9876-8c522df04070" (UID: "d54555c1-98a5-4186-9876-8c522df04070"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:03:18.251797 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d54555c1-98a5-4186-9876-8c522df04070" (UID: "d54555c1-98a5-4186-9876-8c522df04070"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:18.251797 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.251773    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d54555c1-98a5-4186-9876-8c522df04070" (UID: "d54555c1-98a5-4186-9876-8c522df04070"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:18.253462 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.253442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54555c1-98a5-4186-9876-8c522df04070-kube-api-access-9srxc" (OuterVolumeSpecName: "kube-api-access-9srxc") pod "d54555c1-98a5-4186-9876-8c522df04070" (UID: "d54555c1-98a5-4186-9876-8c522df04070"). InnerVolumeSpecName "kube-api-access-9srxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:18.253462 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.253454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54555c1-98a5-4186-9876-8c522df04070-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d54555c1-98a5-4186-9876-8c522df04070" (UID: "d54555c1-98a5-4186-9876-8c522df04070"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:18.261819 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.261794    2563 generic.go:356] "Generic (PLEG): container finished" podID="d54555c1-98a5-4186-9876-8c522df04070" containerID="b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9" exitCode=0
Sep 17 16:03:18.261922 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.261827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerDied","Data":"b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9"}
Sep 17 16:03:18.261922 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.261855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"
Sep 17 16:03:18.261922 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.261872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt" event={"ID":"d54555c1-98a5-4186-9876-8c522df04070","Type":"ContainerDied","Data":"7bcd7645790a34454f18e5190b395b045faf032f6119659295b643d9e22846c0"}
Sep 17 16:03:18.261922 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.261897    2563 scope.go:117] "RemoveContainer" containerID="d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54"
Sep 17 16:03:18.270266 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.270233    2563 scope.go:117] "RemoveContainer" containerID="b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9"
Sep 17 16:03:18.277050 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.277032    2563 scope.go:117] "RemoveContainer" containerID="344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5"
Sep 17 16:03:18.283879 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.283852    2563 scope.go:117] "RemoveContainer" containerID="d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54"
Sep 17 16:03:18.284288 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:03:18.284202    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54\": container with ID starting with d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54 not found: ID does not exist" containerID="d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54"
Sep 17 16:03:18.284382 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.284233    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54"} err="failed to get container status \"d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54\": rpc error: code = NotFound desc = could not find container \"d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54\": container with ID starting with d499ae81708c55c710fe4737b699485260ae1d899f65971ca624163d9ca42d54 not found: ID does not exist"
Sep 17 16:03:18.284382 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.284334    2563 scope.go:117] "RemoveContainer" containerID="b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9"
Sep 17 16:03:18.284684 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:03:18.284665    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9\": container with ID starting with b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9 not found: ID does not exist" containerID="b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9"
Sep 17 16:03:18.284769 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.284690    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9"} err="failed to get container status \"b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9\": rpc error: code = NotFound desc = could not find container \"b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9\": container with ID starting with b0db092ab273cb7e1e64186ed1f6fdadce149b79e9cb255ec2e16bb5a1de8ae9 not found: ID does not exist"
Sep 17 16:03:18.284769 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.284706    2563 scope.go:117] "RemoveContainer" containerID="344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5"
Sep 17 16:03:18.284961 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:03:18.284939    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5\": container with ID starting with 344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5 not found: ID does not exist" containerID="344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5"
Sep 17 16:03:18.285008 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.284966    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5"} err="failed to get container status \"344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5\": rpc error: code = NotFound desc = could not find container \"344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5\": container with ID starting with 344b24ac9a003cde14a18b929fc98244ed05d75f6f086de5a53f2f97a2a79ad5 not found: ID does not exist"
Sep 17 16:03:18.286052 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.286031    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"]
Sep 17 16:03:18.291720 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.291699    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-qp4kt"]
Sep 17 16:03:18.352818 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.352797    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:03:18.352818 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.352818    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54555c1-98a5-4186-9876-8c522df04070-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:03:18.352933 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.352827    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9srxc\" (UniqueName: \"kubernetes.io/projected/d54555c1-98a5-4186-9876-8c522df04070-kube-api-access-9srxc\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:03:18.352933 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.352835    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d54555c1-98a5-4186-9876-8c522df04070-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:03:18.352933 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:18.352843    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d54555c1-98a5-4186-9876-8c522df04070-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.269810 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:19.269753    2563 generic.go:356] "Generic (PLEG): container finished" podID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerID="5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f" exitCode=0
Sep 17 16:03:19.270136 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:19.269821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerDied","Data":"5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f"}
Sep 17 16:03:19.535022 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:19.534933    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54555c1-98a5-4186-9876-8c522df04070" path="/var/lib/kubelet/pods/d54555c1-98a5-4186-9876-8c522df04070/volumes"
Sep 17 16:03:20.277299 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:20.277263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerStarted","Data":"e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606"}
Sep 17 16:03:20.277645 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:20.277326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerStarted","Data":"0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59"}
Sep 17 16:03:20.277645 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:20.277515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:20.301899 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:20.301860    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podStartSLOduration=2.533046743 podStartE2EDuration="6.301845526s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.768798783s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:20.300335451 +0000 UTC m=+1453.262635014" watchObservedRunningTime="2026-09-17 16:03:20.301845526 +0000 UTC m=+1453.264145070"
Sep 17 16:03:21.281704 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:21.281676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:21.282817 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:21.282793    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:03:22.285664 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:22.285613    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:03:27.290059 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:27.290032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:03:27.290559 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:27.290526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:03:37.290769 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:37.290729    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:03:47.290543 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:47.290497    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:03:57.290468 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:03:57.290430    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:04:07.291062 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:07.291023    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:04:17.290696 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:17.290650    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:04:27.290881 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:27.290838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:04:37.291074 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:37.290992    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:04:44.331718 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:44.331684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"]
Sep 17 16:04:44.332194 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:44.331920    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" containerID="cri-o://0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59" gracePeriod=30
Sep 17 16:04:44.332194 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:44.331971    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kube-rbac-proxy" containerID="cri-o://e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606" gracePeriod=30
Sep 17 16:04:44.648760 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:44.648729    2563 generic.go:356] "Generic (PLEG): container finished" podID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerID="e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606" exitCode=2
Sep 17 16:04:44.648879 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:44.648770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerDied","Data":"e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606"}
Sep 17 16:04:47.286201 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:47.286164    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:04:47.290448 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:47.290422    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:04:48.274428 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.274406    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:04:48.472379 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472355    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm28v\" (UniqueName: \"kubernetes.io/projected/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-api-access-lm28v\") pod \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") "
Sep 17 16:04:48.472379 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83eb7707-28fa-4f58-8416-c17fbf7a95f1-proxy-tls\") pod \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") "
Sep 17 16:04:48.472824 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kserve-provision-location\") pod \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") "
Sep 17 16:04:48.472824 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472493    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-rbac-proxy-sar-config\") pod \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") "
Sep 17 16:04:48.472824 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-cabundle-cert\") pod \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\" (UID: \"83eb7707-28fa-4f58-8416-c17fbf7a95f1\") "
Sep 17 16:04:48.472948 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83eb7707-28fa-4f58-8416-c17fbf7a95f1" (UID: "83eb7707-28fa-4f58-8416-c17fbf7a95f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:48.472948 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83eb7707-28fa-4f58-8416-c17fbf7a95f1" (UID: "83eb7707-28fa-4f58-8416-c17fbf7a95f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:48.472948 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.472860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83eb7707-28fa-4f58-8416-c17fbf7a95f1" (UID: "83eb7707-28fa-4f58-8416-c17fbf7a95f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:48.474476 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.474450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-api-access-lm28v" (OuterVolumeSpecName: "kube-api-access-lm28v") pod "83eb7707-28fa-4f58-8416-c17fbf7a95f1" (UID: "83eb7707-28fa-4f58-8416-c17fbf7a95f1"). InnerVolumeSpecName "kube-api-access-lm28v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:48.474476 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.474466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83eb7707-28fa-4f58-8416-c17fbf7a95f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83eb7707-28fa-4f58-8416-c17fbf7a95f1" (UID: "83eb7707-28fa-4f58-8416-c17fbf7a95f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:48.573818 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.573798    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.573818 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.573818    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83eb7707-28fa-4f58-8416-c17fbf7a95f1-cabundle-cert\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.573944 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.573828    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm28v\" (UniqueName: \"kubernetes.io/projected/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kube-api-access-lm28v\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.573944 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.573837    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83eb7707-28fa-4f58-8416-c17fbf7a95f1-proxy-tls\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.573944 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.573848    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83eb7707-28fa-4f58-8416-c17fbf7a95f1-kserve-provision-location\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.668339 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.668317    2563 generic.go:356] "Generic (PLEG): container finished" podID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerID="0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59" exitCode=0
Sep 17 16:04:48.668424 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.668370    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"
Sep 17 16:04:48.668424 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.668395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerDied","Data":"0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59"}
Sep 17 16:04:48.668521 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.668440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6" event={"ID":"83eb7707-28fa-4f58-8416-c17fbf7a95f1","Type":"ContainerDied","Data":"38014b9b49efcaeb2dc96ab1a21642857d295b1717eaa9ee7d3c3d44e7430e12"}
Sep 17 16:04:48.668521 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.668465    2563 scope.go:117] "RemoveContainer" containerID="e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606"
Sep 17 16:04:48.676913 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.676896    2563 scope.go:117] "RemoveContainer" containerID="0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59"
Sep 17 16:04:48.684362 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.684346    2563 scope.go:117] "RemoveContainer" containerID="5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f"
Sep 17 16:04:48.693646 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.693628    2563 scope.go:117] "RemoveContainer" containerID="e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606"
Sep 17 16:04:48.693916 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:04:48.693894    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606\": container with ID starting with e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606 not found: ID does not exist" containerID="e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606"
Sep 17 16:04:48.693979 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.693924    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606"} err="failed to get container status \"e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606\": rpc error: code = NotFound desc = could not find container \"e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606\": container with ID starting with e0df45889105f53ff85a8ae5c5564e14107ba03b4eb17d883ef1f5ce4bac0606 not found: ID does not exist"
Sep 17 16:04:48.693979 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.693946    2563 scope.go:117] "RemoveContainer" containerID="0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59"
Sep 17 16:04:48.694222 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:04:48.694203    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59\": container with ID starting with 0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59 not found: ID does not exist" containerID="0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59"
Sep 17 16:04:48.694222 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.694215    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"]
Sep 17 16:04:48.694376 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.694230    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59"} err="failed to get container status \"0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59\": rpc error: code = NotFound desc = could not find container \"0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59\": container with ID starting with 0988e180f99f64c468aa1d327059f3a52bae582ccd2aeed2a30b50ea89792a59 not found: ID does not exist"
Sep 17 16:04:48.694376 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.694269    2563 scope.go:117] "RemoveContainer" containerID="5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f"
Sep 17 16:04:48.694546 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:04:48.694529    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f\": container with ID starting with 5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f not found: ID does not exist" containerID="5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f"
Sep 17 16:04:48.694586 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.694550    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f"} err="failed to get container status \"5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f\": rpc error: code = NotFound desc = could not find container \"5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f\": container with ID starting with 5348d7c2f11155e477d4a40255c4b26f789b4616c3f38434bb490a69db9b7e3f not found: ID does not exist"
Sep 17 16:04:48.699454 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:48.699436    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-r45v6"]
Sep 17 16:04:49.535195 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:04:49.535156    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" path="/var/lib/kubelet/pods/83eb7707-28fa-4f58-8416-c17fbf7a95f1/volumes"
Sep 17 16:06:54.940735 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.940698    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mkz5/must-gather-tn6zw"]
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941042    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941065    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="storage-initializer"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941071    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="storage-initializer"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kube-rbac-proxy"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941092    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kube-rbac-proxy"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941104    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="storage-initializer"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941111    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="storage-initializer"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941120    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941125    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941138    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kube-rbac-proxy"
Sep 17 16:06:54.941180 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941143    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kube-rbac-proxy"
Sep 17 16:06:54.941673 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kserve-container"
Sep 17 16:06:54.941673 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941204    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kserve-container"
Sep 17 16:06:54.941673 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54555c1-98a5-4186-9876-8c522df04070" containerName="kube-rbac-proxy"
Sep 17 16:06:54.941673 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.941219    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83eb7707-28fa-4f58-8416-c17fbf7a95f1" containerName="kube-rbac-proxy"
Sep 17 16:06:54.944452 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.944432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:54.947235 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.947217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mkz5\"/\"kube-root-ca.crt\""
Sep 17 16:06:54.948266 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.948251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mkz5\"/\"default-dockercfg-b5qvh\""
Sep 17 16:06:54.948266 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.948259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mkz5\"/\"openshift-service-ca.crt\""
Sep 17 16:06:54.953801 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.953781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mkz5/must-gather-tn6zw"]
Sep 17 16:06:54.990720 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.990699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-must-gather-output\") pod \"must-gather-tn6zw\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") " pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:54.990817 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:54.990740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gp9\" (UniqueName: \"kubernetes.io/projected/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-kube-api-access-j8gp9\") pod \"must-gather-tn6zw\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") " pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:55.091943 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.091917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gp9\" (UniqueName: \"kubernetes.io/projected/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-kube-api-access-j8gp9\") pod \"must-gather-tn6zw\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") " pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:55.092043 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.092023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-must-gather-output\") pod \"must-gather-tn6zw\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") " pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:55.092326 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.092307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-must-gather-output\") pod \"must-gather-tn6zw\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") " pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:55.101386 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.101363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gp9\" (UniqueName: \"kubernetes.io/projected/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-kube-api-access-j8gp9\") pod \"must-gather-tn6zw\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") " pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:55.267100 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.267045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:06:55.545329 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.545304    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mkz5/must-gather-tn6zw"]
Sep 17 16:06:55.548014 ip-10-0-143-66 kubenswrapper[2563]: W0917 16:06:55.547988    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ce3eaaf_6029_48c9_b58b_e4ed673f5de7.slice/crio-9376f4238db69f0a21d05e23a940b9b53cf84817fea6c880e5ba9599b88c9bdd WatchSource:0}: Error finding container 9376f4238db69f0a21d05e23a940b9b53cf84817fea6c880e5ba9599b88c9bdd: Status 404 returned error can't find the container with id 9376f4238db69f0a21d05e23a940b9b53cf84817fea6c880e5ba9599b88c9bdd
Sep 17 16:06:55.549942 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:55.549922    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:06:56.223553 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:06:56.223517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mkz5/must-gather-tn6zw" event={"ID":"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7","Type":"ContainerStarted","Data":"9376f4238db69f0a21d05e23a940b9b53cf84817fea6c880e5ba9599b88c9bdd"}
Sep 17 16:07:02.253407 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:02.253368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mkz5/must-gather-tn6zw" event={"ID":"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7","Type":"ContainerStarted","Data":"64ce31511f1ec5da1b27cac897df707381c049334fd2c64813b077bd6a3c1dbf"}
Sep 17 16:07:02.253407 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:02.253413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mkz5/must-gather-tn6zw" event={"ID":"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7","Type":"ContainerStarted","Data":"5402c85e1120af2bf365e71d237d65f9aaf503ea789428103dd823ca033e4d28"}
Sep 17 16:07:02.276003 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:02.275942    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mkz5/must-gather-tn6zw" podStartSLOduration=1.988951594 podStartE2EDuration="8.275925198s" totalImagesPullingTime="6.286973604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:02.275164015 +0000 UTC m=+1675.237463561" watchObservedRunningTime="2026-09-17 16:07:02.275925198 +0000 UTC m=+1675.238224744"
Sep 17 16:07:25.365656 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:25.365620    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerID="5402c85e1120af2bf365e71d237d65f9aaf503ea789428103dd823ca033e4d28" exitCode=0
Sep 17 16:07:25.366090 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:25.365708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mkz5/must-gather-tn6zw" event={"ID":"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7","Type":"ContainerDied","Data":"5402c85e1120af2bf365e71d237d65f9aaf503ea789428103dd823ca033e4d28"}
Sep 17 16:07:25.366090 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:25.366019    2563 scope.go:117] "RemoveContainer" containerID="5402c85e1120af2bf365e71d237d65f9aaf503ea789428103dd823ca033e4d28"
Sep 17 16:07:25.834027 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:25.834002    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8mkz5_must-gather-tn6zw_6ce3eaaf-6029-48c9-b58b-e4ed673f5de7/gather/0.log"
Sep 17 16:07:29.700626 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:29.700588    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-78bdg_6369f6c9-96e2-4131-a71f-a778cee270d2/global-pull-secret-syncer/0.log"
Sep 17 16:07:29.841879 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:29.841849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bpphv_80276513-b7ca-446e-b5c4-db0572ae2be6/konnectivity-agent/0.log"
Sep 17 16:07:29.959209 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:29.959122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-66.ec2.internal_9fbe400037057f056594337561e74444/haproxy/0.log"
Sep 17 16:07:31.264676 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.264630    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8mkz5/must-gather-tn6zw"]
Sep 17 16:07:31.265153 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.264875    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-8mkz5/must-gather-tn6zw" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="copy" containerID="cri-o://64ce31511f1ec5da1b27cac897df707381c049334fd2c64813b077bd6a3c1dbf" gracePeriod=2
Sep 17 16:07:31.270318 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.270286    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8mkz5/must-gather-tn6zw"]
Sep 17 16:07:31.396801 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.396773    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8mkz5_must-gather-tn6zw_6ce3eaaf-6029-48c9-b58b-e4ed673f5de7/copy/0.log"
Sep 17 16:07:31.397322 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.397262    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerID="64ce31511f1ec5da1b27cac897df707381c049334fd2c64813b077bd6a3c1dbf" exitCode=143
Sep 17 16:07:31.496709 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.496686    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8mkz5_must-gather-tn6zw_6ce3eaaf-6029-48c9-b58b-e4ed673f5de7/copy/0.log"
Sep 17 16:07:31.498540 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.498524    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:07:31.500812 ip-10-0-143-66 kubenswrapper[2563]: E0917 16:07:31.500790    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tn6zw\" is forbidden: User \"system:node:ip-10-0-143-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8mkz5\": no relationship found between node 'ip-10-0-143-66.ec2.internal' and this object" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:07:31.602942 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.602889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-must-gather-output\") pod \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") "
Sep 17 16:07:31.603035 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.602982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8gp9\" (UniqueName: \"kubernetes.io/projected/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-kube-api-access-j8gp9\") pod \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\" (UID: \"6ce3eaaf-6029-48c9-b58b-e4ed673f5de7\") "
Sep 17 16:07:31.604956 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.604926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" (UID: "6ce3eaaf-6029-48c9-b58b-e4ed673f5de7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:31.605036 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.605011    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-kube-api-access-j8gp9" (OuterVolumeSpecName: "kube-api-access-j8gp9") pod "6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" (UID: "6ce3eaaf-6029-48c9-b58b-e4ed673f5de7"). InnerVolumeSpecName "kube-api-access-j8gp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:31.704532 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.704501    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gp9\" (UniqueName: \"kubernetes.io/projected/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-kube-api-access-j8gp9\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:07:31.704532 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:31.704530    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7-must-gather-output\") on node \"ip-10-0-143-66.ec2.internal\" DevicePath \"\""
Sep 17 16:07:32.403298 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:32.403268    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8mkz5_must-gather-tn6zw_6ce3eaaf-6029-48c9-b58b-e4ed673f5de7/copy/0.log"
Sep 17 16:07:32.403771 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:32.403747    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mkz5/must-gather-tn6zw"
Sep 17 16:07:32.403771 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:32.403760    2563 scope.go:117] "RemoveContainer" containerID="64ce31511f1ec5da1b27cac897df707381c049334fd2c64813b077bd6a3c1dbf"
Sep 17 16:07:32.413025 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:32.413006    2563 scope.go:117] "RemoveContainer" containerID="5402c85e1120af2bf365e71d237d65f9aaf503ea789428103dd823ca033e4d28"
Sep 17 16:07:33.535855 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.535824    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" path="/var/lib/kubelet/pods/6ce3eaaf-6029-48c9-b58b-e4ed673f5de7/volumes"
Sep 17 16:07:33.741440 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.741417    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/alertmanager/0.log"
Sep 17 16:07:33.773232 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.773212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/config-reloader/0.log"
Sep 17 16:07:33.807463 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.807444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/kube-rbac-proxy-web/0.log"
Sep 17 16:07:33.841178 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.841155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/kube-rbac-proxy/0.log"
Sep 17 16:07:33.870279 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.870262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/kube-rbac-proxy-metric/0.log"
Sep 17 16:07:33.903376 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.903360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/prom-label-proxy/0.log"
Sep 17 16:07:33.934379 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:33.934359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_766f2c5c-47b7-44d1-8dee-45a3a3adaf41/init-config-reloader/0.log"
Sep 17 16:07:34.018367 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.018313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pnsn6_17469123-740f-4a83-a769-c4e98ce51c2b/kube-state-metrics/0.log"
Sep 17 16:07:34.048790 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.048772    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pnsn6_17469123-740f-4a83-a769-c4e98ce51c2b/kube-rbac-proxy-main/0.log"
Sep 17 16:07:34.081763 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.081738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pnsn6_17469123-740f-4a83-a769-c4e98ce51c2b/kube-rbac-proxy-self/0.log"
Sep 17 16:07:34.193855 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.193836    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-4pxms_9aba322d-16d9-4df4-968a-7878d9748414/monitoring-plugin/0.log"
Sep 17 16:07:34.419062 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.419043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxb98_858612e0-c5da-489e-a20d-c53433a8a538/node-exporter/0.log"
Sep 17 16:07:34.449975 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.449957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxb98_858612e0-c5da-489e-a20d-c53433a8a538/kube-rbac-proxy/0.log"
Sep 17 16:07:34.483454 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.483432    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxb98_858612e0-c5da-489e-a20d-c53433a8a538/init-textfile/0.log"
Sep 17 16:07:34.606956 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.606902    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2jc4j_1c619fea-0a0d-4b6b-b6e1-771811e8fc60/kube-rbac-proxy-main/0.log"
Sep 17 16:07:34.641704 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.641686    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2jc4j_1c619fea-0a0d-4b6b-b6e1-771811e8fc60/kube-rbac-proxy-self/0.log"
Sep 17 16:07:34.670024 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.670006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2jc4j_1c619fea-0a0d-4b6b-b6e1-771811e8fc60/openshift-state-metrics/0.log"
Sep 17 16:07:34.717617 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.717590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/prometheus/0.log"
Sep 17 16:07:34.750950 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.750930    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/config-reloader/0.log"
Sep 17 16:07:34.778396 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.778381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/thanos-sidecar/0.log"
Sep 17 16:07:34.810528 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.810511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/kube-rbac-proxy-web/0.log"
Sep 17 16:07:34.844602 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.844580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/kube-rbac-proxy/0.log"
Sep 17 16:07:34.874667 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.874644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/kube-rbac-proxy-thanos/0.log"
Sep 17 16:07:34.903621 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.903603    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f0917d62-1786-4f5c-8fff-42dbf2769408/init-config-reloader/0.log"
Sep 17 16:07:34.945532 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.945509    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-k4cfz_9561363d-ea2b-4f0f-b0d8-808683890341/prometheus-operator/0.log"
Sep 17 16:07:34.967653 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:34.967636    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-k4cfz_9561363d-ea2b-4f0f-b0d8-808683890341/kube-rbac-proxy/0.log"
Sep 17 16:07:35.000161 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.000137    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-chq5w_d981a759-2447-448d-9758-ee3f05f973b4/prometheus-operator-admission-webhook/0.log"
Sep 17 16:07:35.034205 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.034188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55db5d8686-r5lpn_aa46cf83-2bab-47d5-a066-10c7c77474e6/telemeter-client/0.log"
Sep 17 16:07:35.059948 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.059928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55db5d8686-r5lpn_aa46cf83-2bab-47d5-a066-10c7c77474e6/reload/0.log"
Sep 17 16:07:35.092825 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.092806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55db5d8686-r5lpn_aa46cf83-2bab-47d5-a066-10c7c77474e6/kube-rbac-proxy/0.log"
Sep 17 16:07:35.132871 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.132829    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c6c68f5c-rppcb_a37d3c8b-1898-41bd-ad05-3b8d6b1075a4/thanos-query/0.log"
Sep 17 16:07:35.155837 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.155814    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c6c68f5c-rppcb_a37d3c8b-1898-41bd-ad05-3b8d6b1075a4/kube-rbac-proxy-web/0.log"
Sep 17 16:07:35.194717 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.194695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c6c68f5c-rppcb_a37d3c8b-1898-41bd-ad05-3b8d6b1075a4/kube-rbac-proxy/0.log"
Sep 17 16:07:35.234273 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.234229    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c6c68f5c-rppcb_a37d3c8b-1898-41bd-ad05-3b8d6b1075a4/prom-label-proxy/0.log"
Sep 17 16:07:35.267773 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.267744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c6c68f5c-rppcb_a37d3c8b-1898-41bd-ad05-3b8d6b1075a4/kube-rbac-proxy-rules/0.log"
Sep 17 16:07:35.300226 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:35.300202    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c6c68f5c-rppcb_a37d3c8b-1898-41bd-ad05-3b8d6b1075a4/kube-rbac-proxy-metrics/0.log"
Sep 17 16:07:36.587266 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:36.587220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-t78tf_d605099e-24e9-49cb-a936-bc6fc847d771/networking-console-plugin/0.log"
Sep 17 16:07:37.518709 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.518676    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"]
Sep 17 16:07:37.519059 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.519045    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="copy"
Sep 17 16:07:37.519121 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.519061    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="copy"
Sep 17 16:07:37.519121 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.519077    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="gather"
Sep 17 16:07:37.519121 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.519083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="gather"
Sep 17 16:07:37.519229 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.519150    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="gather"
Sep 17 16:07:37.519229 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.519158    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce3eaaf-6029-48c9-b58b-e4ed673f5de7" containerName="copy"
Sep 17 16:07:37.522466 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.522449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.525128 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.525104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nxdvm\"/\"default-dockercfg-mzt7p\""
Sep 17 16:07:37.525285 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.525134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxdvm\"/\"openshift-service-ca.crt\""
Sep 17 16:07:37.526225 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.526208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxdvm\"/\"kube-root-ca.crt\""
Sep 17 16:07:37.535432 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.535409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"]
Sep 17 16:07:37.547557 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.547533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-podres\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.547680 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.547565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-sys\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.547680 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.547597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqkh\" (UniqueName: \"kubernetes.io/projected/c111ff2c-fd0c-4779-bd90-6501a9594c3c-kube-api-access-hdqkh\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.547803 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.547698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-lib-modules\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.547803 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.547752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-proc\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.620359 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.620333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-9b8dh_72288695-a976-4e8e-886f-6b8924db3cf4/download-server/0.log"
Sep 17 16:07:37.648846 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-proc\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.648945 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-podres\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.648945 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-sys\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.648945 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-proc\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.648945 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-sys\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.649112 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqkh\" (UniqueName: \"kubernetes.io/projected/c111ff2c-fd0c-4779-bd90-6501a9594c3c-kube-api-access-hdqkh\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.649112 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-lib-modules\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.649112 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.648996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-podres\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.649112 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.649084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c111ff2c-fd0c-4779-bd90-6501a9594c3c-lib-modules\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.667115 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.667084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqkh\" (UniqueName: \"kubernetes.io/projected/c111ff2c-fd0c-4779-bd90-6501a9594c3c-kube-api-access-hdqkh\") pod \"perf-node-gather-daemonset-tlpq7\" (UID: \"c111ff2c-fd0c-4779-bd90-6501a9594c3c\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:37.833410 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:37.833358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:38.110438 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:38.110416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"]
Sep 17 16:07:38.112614 ip-10-0-143-66 kubenswrapper[2563]: W0917 16:07:38.112587    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc111ff2c_fd0c_4779_bd90_6501a9594c3c.slice/crio-d85dde1d50fe77b2067fa0f0b30b6762743ccfaa926acc22e739cc2f936f9e1d WatchSource:0}: Error finding container d85dde1d50fe77b2067fa0f0b30b6762743ccfaa926acc22e739cc2f936f9e1d: Status 404 returned error can't find the container with id d85dde1d50fe77b2067fa0f0b30b6762743ccfaa926acc22e739cc2f936f9e1d
Sep 17 16:07:38.183183 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:38.183163    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-hssqr_69a4cf25-5cba-4821-a187-6042870523ad/volume-data-source-validator/0.log"
Sep 17 16:07:38.441124 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:38.441098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7" event={"ID":"c111ff2c-fd0c-4779-bd90-6501a9594c3c","Type":"ContainerStarted","Data":"035eb847c25adc65f5f803d1bae85809ab7439407b4be1f0ac60008f5e8c6615"}
Sep 17 16:07:38.441275 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:38.441156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7" event={"ID":"c111ff2c-fd0c-4779-bd90-6501a9594c3c","Type":"ContainerStarted","Data":"d85dde1d50fe77b2067fa0f0b30b6762743ccfaa926acc22e739cc2f936f9e1d"}
Sep 17 16:07:38.441275 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:38.441187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:39.196120 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:39.196093    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4zz5s_0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81/dns/0.log"
Sep 17 16:07:39.225479 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:39.225453    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4zz5s_0d73d927-bcfe-4ed9-9b5d-91e8d86e0d81/kube-rbac-proxy/0.log"
Sep 17 16:07:39.455845 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:39.455782    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmrjn_2d0b166d-7a54-4772-beea-06c71d2a67fe/dns-node-resolver/0.log"
Sep 17 16:07:40.049376 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:40.049346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z8vfk_b9a93a6a-8f86-4e35-83f1-a55ff30da13f/node-ca/0.log"
Sep 17 16:07:40.856726 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:40.856692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-578cf489b4-4hnf6_a6726151-1a0b-4092-a4f3-5356c7d72b22/router/0.log"
Sep 17 16:07:41.237254 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:41.237204    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v97nd_44603772-8a08-4435-a283-dda01bf4bafa/serve-healthcheck-canary/0.log"
Sep 17 16:07:41.779546 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:41.779519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvvp2_a9c30b22-cb0b-40ea-a35a-a45ebaecd115/kube-rbac-proxy/0.log"
Sep 17 16:07:41.806707 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:41.806675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvvp2_a9c30b22-cb0b-40ea-a35a-a45ebaecd115/exporter/0.log"
Sep 17 16:07:41.832990 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:41.832963    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wvvp2_a9c30b22-cb0b-40ea-a35a-a45ebaecd115/extractor/0.log"
Sep 17 16:07:44.455686 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:44.455651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7"
Sep 17 16:07:44.479509 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:44.479451    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-tlpq7" podStartSLOduration=7.4794314 podStartE2EDuration="7.4794314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:38.458928829 +0000 UTC m=+1711.421228389" watchObservedRunningTime="2026-09-17 16:07:44.4794314 +0000 UTC m=+1717.441730948"
Sep 17 16:07:44.484495 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:44.484471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lszj8_c9ad5245-7b7e-4ced-b3f1-0be9be0e481e/s3-init/0.log"
Sep 17 16:07:44.509472 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:44.509449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lszj8_c9ad5245-7b7e-4ced-b3f1-0be9be0e481e/download-hf-model/0.log"
Sep 17 16:07:50.562898 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.562870    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2l64w_b75028fb-ee74-4933-9375-41d26937c9f7/kube-multus/0.log"
Sep 17 16:07:50.625000 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.624974    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:07:50.656256 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.656214    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/egress-router-binary-copy/0.log"
Sep 17 16:07:50.685484 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.685455    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/cni-plugins/0.log"
Sep 17 16:07:50.712636 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.712614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/bond-cni-plugin/0.log"
Sep 17 16:07:50.740976 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.740956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/routeoverride-cni/0.log"
Sep 17 16:07:50.768212 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.768191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/whereabouts-cni-bincopy/0.log"
Sep 17 16:07:50.793942 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:50.793909    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24wtg_e57adf70-aba9-4a36-a5ee-7ffdf90a6110/whereabouts-cni/0.log"
Sep 17 16:07:51.350951 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:51.350932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-stkm5_932cded3-c571-411f-b440-19ac1bd83639/network-metrics-daemon/0.log"
Sep 17 16:07:51.378943 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:51.378921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-stkm5_932cded3-c571-411f-b440-19ac1bd83639/kube-rbac-proxy/0.log"
Sep 17 16:07:52.495084 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.495051    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/ovn-controller/0.log"
Sep 17 16:07:52.516834 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.516810    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/ovn-acl-logging/0.log"
Sep 17 16:07:52.550264 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.550204    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/kube-rbac-proxy-node/0.log"
Sep 17 16:07:52.579459 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.579441    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:07:52.606206 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.606185    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/northd/0.log"
Sep 17 16:07:52.633427 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.633410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/nbdb/0.log"
Sep 17 16:07:52.667942 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.667927    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/sbdb/0.log"
Sep 17 16:07:52.731462 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:52.731443    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5hdl_b2aee588-9d72-44b7-99be-919ddc5cec56/ovnkube-controller/0.log"
Sep 17 16:07:54.161188 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:54.161160    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xpcqf_d7fbee30-abee-40da-9b15-9602c5e4195c/network-check-target-container/0.log"
Sep 17 16:07:55.261725 ip-10-0-143-66 kubenswrapper[2563]: I0917 16:07:55.261695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-hjvd2_134bc838-5236-4aee-adc0-05b1b7d57fde/iptables-alerter/0.log"