Sep 17 18:36:49.929906 ip-10-0-136-86 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:36:49.929918 ip-10-0-136-86 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:36:49.929927 ip-10-0-136-86 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:36:49.930137 ip-10-0-136-86 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:37:00.179060 ip-10-0-136-86 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:37:00.179079 ip-10-0-136-86 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5f76aa014805471db76d1311a10cc965 --
Sep 17 18:38:41.585172 ip-10-0-136-86 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:38:45.006759 ip-10-0-136-86 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:45.006759 ip-10-0-136-86 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:38:45.006759 ip-10-0-136-86 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:45.006759 ip-10-0-136-86 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:38:45.006759 ip-10-0-136-86 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:45.007619 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.006814    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:38:45.010299 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010281    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:45.010299 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010298    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:45.010299 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010303    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010306    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010310    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010314    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010317    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010320    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010323    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010326    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010329    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010332    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010334    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010337    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010340    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010343    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010346    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010349    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010354    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010357    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010361    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:45.010396 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010364    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010371    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010374    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010378    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010381    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010384    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010387    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010390    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010393    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010396    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010399    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010402    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010405    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010408    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010413    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010417    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010419    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010422    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010425    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010427    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:45.010874 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010431    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010433    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010436    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010440    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010443    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010447    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010449    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010452    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010454    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010457    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010460    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010462    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010465    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010467    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010471    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010474    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010477    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010480    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010483    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:45.011374 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010486    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010490    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010494    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010497    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010500    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010503    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010506    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010509    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010512    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010514    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010517    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010521    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010524    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010527    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010529    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010532    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010535    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010538    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010540    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010543    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:45.011859 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010545    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010548    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010551    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010554    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010557    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010559    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010562    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010565    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010568    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010571    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010574    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010576    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010579    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010581    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010584    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010588    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010590    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010593    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.010596    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011383    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:45.012353 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011392    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011396    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011401    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011405    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011409    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011412    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011415    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011418    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011421    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011424    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011426    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011429    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011432    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011434    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011437    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011440    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011443    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011446    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011449    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:45.012947 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011452    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011455    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011458    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011460    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011463    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011466    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011469    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011471    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011474    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011476    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011479    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011482    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011484    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011488    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011492    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011495    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011499    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011502    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011505    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:45.013446 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011508    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011511    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011514    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011517    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011519    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011522    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011525    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011528    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011531    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011534    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011536    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011539    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011542    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011545    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011547    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011550    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011553    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011555    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011558    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011561    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:45.013933 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011563    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011566    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011569    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011571    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011574    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011577    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011580    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011582    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011585    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011588    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011590    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011593    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011595    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011598    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011601    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011604    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011606    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011609    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011611    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011614    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:45.014458 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011616    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011621    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011624    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011627    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011629    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011633    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011637    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011641    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011644    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011647    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011650    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011653    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011657    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011660    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011662    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011665    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011668    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011670    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011673    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.011675    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:45.014965 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012064    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012070    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012074    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012077    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012082    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012086    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012090    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012093    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012096    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012098    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012101    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012103    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012106    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012111    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012114    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012117    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012120    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012122    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012125    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:45.015469 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012127    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012130    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012132    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012135    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012138    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012140    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012143    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012146    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012150    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012152    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012155    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012158    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012160    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012163    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012166    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012168    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012171    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012174    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012177    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012179    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:45.015999 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012183    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012185    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012188    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012190    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012193    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012195    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012200    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012203    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012206    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012208    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012211    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012214    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012216    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012219    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012222    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012224    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012227    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012229    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012232    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012234    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:45.016497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012238    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012240    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012242    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012245    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012248    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012250    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012253    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012256    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012258    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012261    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012263    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012266    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012269    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012272    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012274    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012277    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012279    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012282    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012287    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:45.017066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012289    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012293    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012297    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012300    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012303    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012307    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012311    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012314    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012316    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012319    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012322    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012324    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012327    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012330    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012332    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012335    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012337    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012340    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012342    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:45.017565 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012347    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.012350    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012426    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012472    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012482    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012487    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012493    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012496    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012500    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012504    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012507    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012510    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012515    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012519    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012522    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012525    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012528    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012531    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012534    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012537    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012541    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012543    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012547    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012550    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:38:45.018052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012553    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012557    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012559    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012563    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012566    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012569    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012572    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012575    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012580    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012583    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012587    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012590    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012593    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012596    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012599    2566 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012607    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012615    2566 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012619    2566 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012622    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012625    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012628    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012632    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012635    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012638    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012641    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:38:45.018641 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012644    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012647    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012650    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012653    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012656    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012659    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012662    2566 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012666    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012669    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012672    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012675    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012678    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012681    2566 flags.go:64] FLAG: --help="false"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012684    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012687    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012692    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012695    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012699    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012702    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012705    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012709    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012712    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012715    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:38:45.019272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012719    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012722    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012726    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012729    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012731    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012734    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012737    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012740    2566 flags.go:64] FLAG: --lock-file=""
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012743    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012746    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012749    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012755    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012758    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012761    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012764    2566 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012767    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012770    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012773    2566 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012777    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012782    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012785    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012789    2566 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012792    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012795    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012803    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:38:45.019887 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012806    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012810    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012813    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012816    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012823    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012840    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012843    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012846    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012850    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012855    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012859    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012862    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012865    2566 flags.go:64] FLAG: --port="10250"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012868    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012871    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b001f043b80c0d3f"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012874    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012877    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012880    2566 flags.go:64] FLAG: --register-node="true"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012883    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012887    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012890    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012892    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012895    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012899    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012902    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:38:45.020501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012905    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012908    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012911    2566 flags.go:64] FLAG: --runonce="false"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012914    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012917    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012920    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012924    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012927    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012930    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012933    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012936    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012941    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012944    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012947    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012950    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012953    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012957    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012961    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012966    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012969    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012973    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012976    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012979    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012982    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012985    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:38:45.021161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012988    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012991    2566 flags.go:64] FLAG: --v="2"
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.012995    2566 flags.go:64] FLAG: --version="false"
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.013000    2566 flags.go:64] FLAG: --vmodule=""
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.013003    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.013006    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013134    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013138    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013141    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013144    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013147    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013152    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013155    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013159    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013162    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013164    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013167    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013170    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013175    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013177    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013180    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:45.021782 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013183    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013186    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013188    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013192    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013194    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013197    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013200    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013203    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013205    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013208    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013211    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013214    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013216    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013219    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013222    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013224    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013227    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013229    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013232    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013235    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:45.022307 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013238    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013240    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013243    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013245    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013249    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013253    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013256    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013258    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013261    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013266    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013269    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013271    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013274    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013277    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013279    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013282    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013292    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013295    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013298    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013303    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:45.022810 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013306    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013309    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013312    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013315    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013317    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013320    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013323    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013326    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013328    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013331    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013333    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013336    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013339    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013342    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013344    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013347    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013349    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013352    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013355    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013358    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:45.023736 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013361    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013366    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013370    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013373    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013376    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013379    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013382    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013385    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013389    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013393    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013396    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013398    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013401    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013403    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013406    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013409    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013412    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013414    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013417    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:45.024422 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013420    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013423    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013425    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013428    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.013431    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.014016    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.021105    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.021122    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.021149    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.021154    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021209    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021214    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021218    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021221    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021224    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:45.024951 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021227    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021231    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021234    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021236    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021239    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021242    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021245    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021248    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021251    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021253    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021256    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021259    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021262    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021264    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021267    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021270    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021273    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021275    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021278    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:45.025327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021280    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021283    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021286    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021288    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021291    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021295    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021298    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021300    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021303    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021305    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021308    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021311    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021314    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021316    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021319    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021322    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021325    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021327    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021330    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021333    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:45.025800 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021336    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021339    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021342    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021344    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021347    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021351    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021355    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021358    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021361    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021364    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021368    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021371    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021374    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021377    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021381    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021385    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021388    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021391    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021394    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:45.026317 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021398    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021401    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021404    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021407    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021409    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021412    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021415    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021418    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021420    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021423    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021426    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021428    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021431    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021433    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021436    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021439    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021441    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021444    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021446    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021449    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:45.026815 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021452    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021454    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021458    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021461    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021464    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021467    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021469    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021472    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021474    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021477    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021480    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021482    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021485    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021489    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021493    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:45.027327 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021496    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.021501    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021634    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021639    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021642    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021645    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021648    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021651    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021654    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021657    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021660    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021663    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021666    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021669    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021671    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:45.027702 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021674    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021677    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021679    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021682    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021685    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021687    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021692    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021695    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021697    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021700    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021703    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021705    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021708    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021712    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021716    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021719    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021721    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021724    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021727    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:45.028089 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021729    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021732    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021734    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021738    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021741    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021744    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021747    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021750    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021753    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021756    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021759    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021762    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021764    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021767    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021770    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021772    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021776    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021778    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021781    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:45.028549 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021784    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021788    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021791    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021793    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021796    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021799    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021802    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021804    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021807    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021809    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021813    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021815    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021818    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021820    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021823    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021843    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021849    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021855    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021858    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021861    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:45.029013 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021864    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021867    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021870    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021872    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021875    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021878    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021880    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021883    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021886    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021889    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021892    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021894    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021897    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021900    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021903    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021905    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021908    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021911    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021914    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021917    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:45.029497 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021919    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021922    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021925    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021928    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021931    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021934    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021936    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:45.021939    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.021944    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.022054    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.025608    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.026323    2566 server.go:1019] "Starting client certificate rotation"
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.026420    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:38:45.030153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.026454    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:38:45.030515 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.028412    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:38:45.030515 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.029561    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:38:45.032930 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.032911    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:38:45.034938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.034921    2566 log.go:25] "Validated CRI v1 image API"
Sep 17 18:38:45.040169 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.040154    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:38:45.043393 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.043362    2566 fs.go:135] Filesystem UUIDs: map[4dcbf627-6eb3-494e-a568-99f757a5bacb:/dev/nvme0n1p4 74eb3ba6-50aa-41cb-89be-a014955d0723:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 18:38:45.043474 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.043391    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:38:45.047694 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.047553    2566 manager.go:216] Machine: {Timestamp:2026-09-17 18:38:45.047069962 +0000 UTC m=+1.899774197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101678 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21459d88e667e39c86a3f3775fba85 SystemUUID:ec21459d-88e6-67e3-9c86-a3f3775fba85 BootID:5f76aa01-4805-471d-b76d-1311a10cc965 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:04:53:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:04:53:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:4a:06:a9:fb:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:38:45.047694 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.047690    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:38:45.047810 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.047763    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:38:45.049122 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.049091    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:38:45.049260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.049124    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-86.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:38:45.050391 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.050377    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:38:45.050436 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.050393    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:38:45.050436 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.050401    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:38:45.050436 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.050411    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:38:45.050517 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.050477    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:38:45.054285 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054273    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:38:45.054392 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054383    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:38:45.054427 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054417    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:38:45.054427 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054426    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:38:45.054484 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054442    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:38:45.054484 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054452    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:38:45.054484 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.054460    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:38:45.055746 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.055603    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:38:45.055746 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.055639    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:38:45.057587 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.057561    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:38:45.059780 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.059764    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:38:45.059874 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.059802    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:38:45.060758 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.060737    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:38:45.061014 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061000    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061021    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061027    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061033    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061043    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061053    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061068    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061073    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:38:45.061081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061081    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:38:45.061288 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061091    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:38:45.061288 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061101    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:38:45.061288 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061106    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:38:45.061288 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061135    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:38:45.061288 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.061140    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:38:45.064864 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.064850    2566 server.go:1268] "Started kubelet"
Sep 17 18:38:45.064978 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.064955    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:38:45.065031 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.064960    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:38:45.065082 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.065031    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:38:45.065952 ip-10-0-136-86 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:38:45.066459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.066188    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:38:45.066580 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.066551    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:38:45.066860 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.066804    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-86.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:38:45.067310 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.067277    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:38:45.067310 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.067277    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:38:45.068912 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.068885    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:38:45.071991 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.071969    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:38:45.072054 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.071977    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:38:45.072120 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.072109    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.072606 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072542    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:38:45.072606 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072550    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:38:45.072606 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072566    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:38:45.072780 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072663    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:38:45.072780 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072676    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:38:45.072867 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072818    2566 factory.go:55] Registering systemd factory
Sep 17 18:38:45.072867 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.072849    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:38:45.073088 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.073074    2566 factory.go:153] Registering CRI-O factory
Sep 17 18:38:45.073125 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.073093    2566 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:38:45.073170 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.073158    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:38:45.073201 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.073185    2566 factory.go:103] Registering Raw factory
Sep 17 18:38:45.073201 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.073200    2566 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:38:45.073780 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.073755    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:38:45.073780 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.073773    2566 manager.go:349] Starting recovery of all containers
Sep 17 18:38:45.073953 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.073914    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:38:45.100099 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.100066    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-snr8n"
Sep 17 18:38:45.100196 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.100100    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-snr8n"
Sep 17 18:38:45.107633 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.107475    2566 manager.go:354] Recovery completed
Sep 17 18:38:45.111554 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.111541    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.113425 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.113412    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.113482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.113439    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.113482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.113449    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.114038 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114024    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:38:45.114038 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114038    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:38:45.114160 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114056    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:38:45.114160 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114093    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.114777 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114760    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.114870 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114790    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.114870 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.114801    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.116147 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.116135    2566 policy_none.go:49] "None policy: Start"
Sep 17 18:38:45.116205 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.116152    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:38:45.116205 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.116165    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:38:45.118244 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.118231    2566 policy_none.go:47] "Start"
Sep 17 18:38:45.156396 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156382    2566 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.156413    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156422    2566 server.go:87] "Starting device plugin registration server"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156591    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156611    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156697    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156772    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.156781    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.157198    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:38:45.169410 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.157239    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.191750 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.191730    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:38:45.192790 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.192776    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:38:45.192885 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.192825    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:38:45.192885 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.192863    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:38:45.193036 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.192905    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:38:45.195036 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.195016    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:45.257154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.257107    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.257901 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.257881    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.257977 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.257910    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.257977 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.257922    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.257977 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.257945    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.266502 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.266490    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.266554 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.266507    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-86.ec2.internal\": node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.285182 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.285161    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.293458 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.293431    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"]
Sep 17 18:38:45.293516 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.293493    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.295124 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.295109    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.295180 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.295137    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.295180 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.295146    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.297393 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.297381    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.297499 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.297486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.297535 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.297522    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.298129 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.298112    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.298214 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.298144    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.298214 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.298157    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.298214 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.298165    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.298214 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.298192    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.298214 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.298205    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.300409 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.300393    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.300471 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.300427    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:45.301168 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.301150    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:45.301260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.301178    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:45.301260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.301190    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:45.324427 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.324413    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-86.ec2.internal\" not found" node="ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.328711 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.328695    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-86.ec2.internal\" not found" node="ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.374180 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.374157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a47bb08de40580886c0a61ec0484439d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal\" (UID: \"a47bb08de40580886c0a61ec0484439d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.374260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.374181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a47bb08de40580886c0a61ec0484439d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal\" (UID: \"a47bb08de40580886c0a61ec0484439d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.374260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.374199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/feb5980f217da1fed6de338fd44f47c5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-86.ec2.internal\" (UID: \"feb5980f217da1fed6de338fd44f47c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.385458 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.385437    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.475210 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.475175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a47bb08de40580886c0a61ec0484439d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal\" (UID: \"a47bb08de40580886c0a61ec0484439d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.475325 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.475214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a47bb08de40580886c0a61ec0484439d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal\" (UID: \"a47bb08de40580886c0a61ec0484439d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.475325 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.475241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/feb5980f217da1fed6de338fd44f47c5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-86.ec2.internal\" (UID: \"feb5980f217da1fed6de338fd44f47c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.475325 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.475270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a47bb08de40580886c0a61ec0484439d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal\" (UID: \"a47bb08de40580886c0a61ec0484439d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.475325 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.475286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/feb5980f217da1fed6de338fd44f47c5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-86.ec2.internal\" (UID: \"feb5980f217da1fed6de338fd44f47c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.475325 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.475287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a47bb08de40580886c0a61ec0484439d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal\" (UID: \"a47bb08de40580886c0a61ec0484439d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.486308 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.486287    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.587068 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.587010    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.628192 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.628169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.632124 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:45.632100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:45.687736 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.687712    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.788184 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.788162    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.888760 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.888704    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:45.989178 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:45.989157    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.026361 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.026337    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:38:46.026883 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.026467    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:38:46.089877 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:46.089856    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.106019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.105990    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:33:45 +0000 UTC" deadline="2028-04-18 12:26:20.686411133 +0000 UTC"
Sep 17 18:38:46.106019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.106016    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13889h47m34.580397531s"
Sep 17 18:38:46.174967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.174946    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:46.190297 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:46.190270    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.213170 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:46.213127    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47bb08de40580886c0a61ec0484439d.slice/crio-e052bf4976ce68518e40ff84ae221ccdb9e4e29b2701402ffd597af921853698 WatchSource:0}: Error finding container e052bf4976ce68518e40ff84ae221ccdb9e4e29b2701402ffd597af921853698: Status 404 returned error can't find the container with id e052bf4976ce68518e40ff84ae221ccdb9e4e29b2701402ffd597af921853698
Sep 17 18:38:46.214562 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:46.214533    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb5980f217da1fed6de338fd44f47c5.slice/crio-563e6a51c65981e32b5a505b35c9fb397e73fab7879759457baf219e3cf3e493 WatchSource:0}: Error finding container 563e6a51c65981e32b5a505b35c9fb397e73fab7879759457baf219e3cf3e493: Status 404 returned error can't find the container with id 563e6a51c65981e32b5a505b35c9fb397e73fab7879759457baf219e3cf3e493
Sep 17 18:38:46.218445 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.218431    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:38:46.290742 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:46.290712    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.391187 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:46.391153    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.435723 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.435673    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:46.491953 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:46.491933    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.592748 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:46.592727    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-86.ec2.internal\" not found"
Sep 17 18:38:46.650967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.650947    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:46.672898 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.672875    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:46.684277 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.684258    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:38:46.685061 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.685039    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal"
Sep 17 18:38:46.691259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:46.691203    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:38:47.056453 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.056386    2566 apiserver.go:52] "Watching apiserver"
Sep 17 18:38:47.063080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.063052    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:38:47.063411 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.063384    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lkmvt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs","openshift-cluster-node-tuning-operator/tuned-xb28h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal","openshift-multus/multus-additional-cni-plugins-wj95n","kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal","openshift-dns/node-resolver-q9dbd","openshift-image-registry/node-ca-qr8gm","openshift-multus/multus-qtzwq","openshift-multus/network-metrics-daemon-9gtzs","openshift-network-diagnostics/network-check-target-qfzqm","openshift-network-operator/iptables-alerter-zjhmb","openshift-ovn-kubernetes/ovnkube-node-xwfls","kube-system/global-pull-secret-syncer-bvtmd"]
Sep 17 18:38:47.065943 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.065924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.066062 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.066000    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:47.068221 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.068196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.070346 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.070329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.071949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.071928    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:38:47.072041 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.071948    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:38:47.072041 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.071962    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.072041 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.071963    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072482    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072541    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5lslv\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072550    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072589    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072661    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072665    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cv5k9\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072789    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.072961 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.072909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.073366 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.072971    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:38:47.075066 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.075047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.077149 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.077130    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.077687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.077669    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-frt59\""
Sep 17 18:38:47.077843 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.077814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.077919 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.077904    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:38:47.078210 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.078192    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:38:47.079767 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.078724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.079767 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.078989    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:38:47.080249 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.080051    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.080747 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.080504    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.080747 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.080638    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4zpkd\""
Sep 17 18:38:47.081427 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.081407    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:38:47.082044 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.082026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.084108 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:38:47.084213 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-kubelet\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084264 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-run-netns\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084264 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-var-lib-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084363 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-etc-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084363 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq98k\" (UniqueName: \"kubernetes.io/projected/b0c1dd66-c309-436d-838a-f3b29ea4af63-kube-api-access-cq98k\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084363 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-sys\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.084483 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrls\" (UniqueName: \"kubernetes.io/projected/6c8dd36c-edaa-439a-a035-815b938708bd-kube-api-access-rwrls\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.084483 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-systemd-units\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084483 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084424    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bqs6z\""
Sep 17 18:38:47.084483 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-slash\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084483 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-ovn\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-cni-bin\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-cni-netd\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8dd36c-edaa-439a-a035-815b938708bd-tmp\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/853ece52-4596-4c55-83af-da02fc283b14-kubelet-config\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/853ece52-4596-4c55-83af-da02fc283b14-dbus\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.084718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-node-log\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-cni-binary-copy\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084814    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysctl-d\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-run\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-host\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.084989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwbg\" (UniqueName: \"kubernetes.io/projected/1e4b8718-ff20-4f8b-bfee-853b19603fdc-kube-api-access-qnwbg\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.085080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-system-cni-dir\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovnkube-config\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-env-overrides\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovnkube-script-lib\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-modprobe-d\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-log-socket\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovn-node-metrics-cert\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysconfig\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-systemd\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-var-lib-kubelet\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c8dd36c-edaa-439a-a035-815b938708bd-etc-tuned\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqpp\" (UniqueName: \"kubernetes.io/projected/e46de969-dafc-495c-a1df-6c72a069ff78-kube-api-access-rvqpp\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-cnibin\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-systemd\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.085616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-kubernetes\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.086209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysctl-conf\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.086209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.085529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-lib-modules\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.086861 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.086842    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.087279 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.087258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzsgn\""
Sep 17 18:38:47.087372 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.087293    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.087430 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.087383    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.089349 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.089039    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:38:47.089349 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.089082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-czbvz\""
Sep 17 18:38:47.089349 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.089199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.089534 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.089377    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.089534 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.089480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.091811 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.091792    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-blmv2\""
Sep 17 18:38:47.091956 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.091935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:38:47.092157 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.092141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:38:47.092223 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.092198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:47.092281 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.092250    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:47.094788 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.094772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.097059 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.097026    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:38:47.097252 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.097208    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p99h4\""
Sep 17 18:38:47.097610 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.097417    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:38:47.097610 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.097151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:38:47.111688 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.111668    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ggqv"
Sep 17 18:38:47.117718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.117698    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ggqv"
Sep 17 18:38:47.173344 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.173318    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:38:47.186260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgsr\" (UniqueName: \"kubernetes.io/projected/79480ed8-4de0-47ef-80ba-80f25ce854f1-kube-api-access-6fgsr\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.186370 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-host\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.186370 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-k8s-cni-cncf-io\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.186370 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/974224b4-09ad-4737-8b71-3df6f66f0dfd-host\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.186370 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.186370 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-host\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovnkube-script-lib\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-systemd\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysconfig\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-cnibin\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.186485    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-systemd\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysconfig\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-cni-bin\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.186549    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:38:47.686532578 +0000 UTC m=+4.539236814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-cni-multus\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.186621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-hostroot\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-conf-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-registration-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-cnibin\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-systemd\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysctl-conf\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysctl-conf\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-lib-modules\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.186986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-run-netns\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-systemd\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-cnibin\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-var-lib-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-run-netns\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovnkube-script-lib\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-etc-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-var-lib-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq98k\" (UniqueName: \"kubernetes.io/projected/b0c1dd66-c309-436d-838a-f3b29ea4af63-kube-api-access-cq98k\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-lib-modules\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-sys\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrls\" (UniqueName: \"kubernetes.io/projected/6c8dd36c-edaa-439a-a035-815b938708bd-kube-api-access-rwrls\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-sys-fs\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-etc-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-multus-certs\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-systemd-units\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-ovn\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-sys\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8dd36c-edaa-439a-a035-815b938708bd-tmp\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-systemd-units\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-ovn\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-socket-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-netns\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-iptables-alerter-script\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.187975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/974224b4-09ad-4737-8b71-3df6f66f0dfd-serviceca\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-cni-binary-copy\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-run-openvswitch\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187745    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-run\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-run\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.187973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.188009    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-socket-dir-parent\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.188101    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:47.68808424 +0000 UTC m=+4.540788477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-cni-binary-copy\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwbg\" (UniqueName: \"kubernetes.io/projected/1e4b8718-ff20-4f8b-bfee-853b19603fdc-kube-api-access-qnwbg\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-system-cni-dir\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.188783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e46de969-dafc-495c-a1df-6c72a069ff78-system-cni-dir\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovnkube-config\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-env-overrides\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-modprobe-d\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c8dd36c-edaa-439a-a035-815b938708bd-etc-tuned\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79480ed8-4de0-47ef-80ba-80f25ce854f1-tmp-dir\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-log-socket\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovn-node-metrics-cert\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-var-lib-kubelet\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-cni-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-device-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqpp\" (UniqueName: \"kubernetes.io/projected/e46de969-dafc-495c-a1df-6c72a069ff78-kube-api-access-rvqpp\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-kubernetes\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-cni-binary-copy\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-daemon-config\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-etc-kubernetes\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.189582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e8be8b1-3cb1-4773-ac22-e32387848821-agent-certs\") pod \"konnectivity-agent-lkmvt\" (UID: \"2e8be8b1-3cb1-4773-ac22-e32387848821\") " pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-env-overrides\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkxz\" (UniqueName: \"kubernetes.io/projected/974224b4-09ad-4737-8b71-3df6f66f0dfd-kube-api-access-xdkxz\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovnkube-config\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-kubelet\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-kubelet\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-host-slash\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-kubernetes\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-slash\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-cni-bin\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e8be8b1-3cb1-4773-ac22-e32387848821-konnectivity-ca\") pod \"konnectivity-agent-lkmvt\" (UID: \"2e8be8b1-3cb1-4773-ac22-e32387848821\") " pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-var-lib-kubelet\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-cni-netd\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.188994    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-modprobe-d\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/853ece52-4596-4c55-83af-da02fc283b14-kubelet-config\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/853ece52-4596-4c55-83af-da02fc283b14-dbus\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hl8g\" (UniqueName: \"kubernetes.io/projected/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-kube-api-access-8hl8g\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.190469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-slash\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9kq\" (UniqueName: \"kubernetes.io/projected/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-kube-api-access-zx9kq\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79480ed8-4de0-47ef-80ba-80f25ce854f1-hosts-file\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-node-log\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysctl-d\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchs7\" (UniqueName: \"kubernetes.io/projected/3692963f-c4d0-4368-b141-5f116a50e509-kube-api-access-lchs7\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-system-cni-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-cni-bin\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/853ece52-4596-4c55-83af-da02fc283b14-kubelet-config\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-node-log\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-cni-netd\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-host-kubelet\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/853ece52-4596-4c55-83af-da02fc283b14-dbus\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.191259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0c1dd66-c309-436d-838a-f3b29ea4af63-log-socket\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.192154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.189704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c8dd36c-edaa-439a-a035-815b938708bd-etc-sysctl-d\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.192154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.190116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e46de969-dafc-495c-a1df-6c72a069ff78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.192154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.190933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8dd36c-edaa-439a-a035-815b938708bd-tmp\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.192154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.191068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0c1dd66-c309-436d-838a-f3b29ea4af63-ovn-node-metrics-cert\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.192154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.191495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c8dd36c-edaa-439a-a035-815b938708bd-etc-tuned\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.197241 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.197155    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:47.198024 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.197681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal" event={"ID":"a47bb08de40580886c0a61ec0484439d","Type":"ContainerStarted","Data":"e052bf4976ce68518e40ff84ae221ccdb9e4e29b2701402ffd597af921853698"}
Sep 17 18:38:47.199042 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.199017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwbg\" (UniqueName: \"kubernetes.io/projected/1e4b8718-ff20-4f8b-bfee-853b19603fdc-kube-api-access-qnwbg\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.199433 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.199415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqpp\" (UniqueName: \"kubernetes.io/projected/e46de969-dafc-495c-a1df-6c72a069ff78-kube-api-access-rvqpp\") pod \"multus-additional-cni-plugins-wj95n\" (UID: \"e46de969-dafc-495c-a1df-6c72a069ff78\") " pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.199510 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.199494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq98k\" (UniqueName: \"kubernetes.io/projected/b0c1dd66-c309-436d-838a-f3b29ea4af63-kube-api-access-cq98k\") pod \"ovnkube-node-xwfls\" (UID: \"b0c1dd66-c309-436d-838a-f3b29ea4af63\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.199726 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.199706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal" event={"ID":"feb5980f217da1fed6de338fd44f47c5","Type":"ContainerStarted","Data":"563e6a51c65981e32b5a505b35c9fb397e73fab7879759457baf219e3cf3e493"}
Sep 17 18:38:47.200295 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.200279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrls\" (UniqueName: \"kubernetes.io/projected/6c8dd36c-edaa-439a-a035-815b938708bd-kube-api-access-rwrls\") pod \"tuned-xb28h\" (UID: \"6c8dd36c-edaa-439a-a035-815b938708bd\") " pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.290135 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-cnibin\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290284 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-cni-bin\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290284 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-cni-multus\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290284 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-cnibin\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290449 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-cni-bin\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290449 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-cni-multus\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290449 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-hostroot\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290449 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-conf-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290449 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-registration-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-sys-fs\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-multus-certs\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-conf-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-hostroot\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-socket-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-netns\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-sys-fs\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-iptables-alerter-script\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/974224b4-09ad-4737-8b71-3df6f66f0dfd-serviceca\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-registration-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-socket-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-socket-dir-parent\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.290683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79480ed8-4de0-47ef-80ba-80f25ce854f1-tmp-dir\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-multus-certs\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-netns\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-cni-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-cni-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-device-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-cni-binary-copy\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-daemon-config\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-etc-kubernetes\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e8be8b1-3cb1-4773-ac22-e32387848821-agent-certs\") pod \"konnectivity-agent-lkmvt\" (UID: \"2e8be8b1-3cb1-4773-ac22-e32387848821\") " pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkxz\" (UniqueName: \"kubernetes.io/projected/974224b4-09ad-4737-8b71-3df6f66f0dfd-kube-api-access-xdkxz\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.290997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-kubelet\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-host-slash\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e8be8b1-3cb1-4773-ac22-e32387848821-konnectivity-ca\") pod \"konnectivity-agent-lkmvt\" (UID: \"2e8be8b1-3cb1-4773-ac22-e32387848821\") " pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hl8g\" (UniqueName: \"kubernetes.io/projected/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-kube-api-access-8hl8g\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291082    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-iptables-alerter-script\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:47.291357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/974224b4-09ad-4737-8b71-3df6f66f0dfd-serviceca\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9kq\" (UniqueName: \"kubernetes.io/projected/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-kube-api-access-zx9kq\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-socket-dir-parent\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79480ed8-4de0-47ef-80ba-80f25ce854f1-hosts-file\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchs7\" (UniqueName: \"kubernetes.io/projected/3692963f-c4d0-4368-b141-5f116a50e509-kube-api-access-lchs7\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-system-cni-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgsr\" (UniqueName: \"kubernetes.io/projected/79480ed8-4de0-47ef-80ba-80f25ce854f1-kube-api-access-6fgsr\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-k8s-cni-cncf-io\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/974224b4-09ad-4737-8b71-3df6f66f0dfd-host\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/974224b4-09ad-4737-8b71-3df6f66f0dfd-host\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79480ed8-4de0-47ef-80ba-80f25ce854f1-hosts-file\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3692963f-c4d0-4368-b141-5f116a50e509-device-dir\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-system-cni-dir\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-var-lib-kubelet\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79480ed8-4de0-47ef-80ba-80f25ce854f1-tmp-dir\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.292796 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.291518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-host-run-k8s-cni-cncf-io\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292796 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.292340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-multus-daemon-config\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292796 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.292629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-host-slash\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.292796 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.292723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-cni-binary-copy\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.292796 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.292762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-etc-kubernetes\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.293050 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.292997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e8be8b1-3cb1-4773-ac22-e32387848821-konnectivity-ca\") pod \"konnectivity-agent-lkmvt\" (UID: \"2e8be8b1-3cb1-4773-ac22-e32387848821\") " pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.295638 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.295176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e8be8b1-3cb1-4773-ac22-e32387848821-agent-certs\") pod \"konnectivity-agent-lkmvt\" (UID: \"2e8be8b1-3cb1-4773-ac22-e32387848821\") " pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.299512 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.299492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9kq\" (UniqueName: \"kubernetes.io/projected/6249ccd6-6c9d-43d6-ae9d-61beefa0e564-kube-api-access-zx9kq\") pod \"iptables-alerter-zjhmb\" (UID: \"6249ccd6-6c9d-43d6-ae9d-61beefa0e564\") " pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.302508 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.302442    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:47.302508 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.302470    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:47.302508 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.302483    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:47.302721 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.302546    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:47.802529474 +0000 UTC m=+4.655233703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:47.304915 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.304893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgsr\" (UniqueName: \"kubernetes.io/projected/79480ed8-4de0-47ef-80ba-80f25ce854f1-kube-api-access-6fgsr\") pod \"node-resolver-q9dbd\" (UID: \"79480ed8-4de0-47ef-80ba-80f25ce854f1\") " pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.305000 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.304946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hl8g\" (UniqueName: \"kubernetes.io/projected/fc3335f9-1553-4f57-b95a-3cd8ac5069e3-kube-api-access-8hl8g\") pod \"multus-qtzwq\" (UID: \"fc3335f9-1553-4f57-b95a-3cd8ac5069e3\") " pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.305202 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.305178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkxz\" (UniqueName: \"kubernetes.io/projected/974224b4-09ad-4737-8b71-3df6f66f0dfd-kube-api-access-xdkxz\") pod \"node-ca-qr8gm\" (UID: \"974224b4-09ad-4737-8b71-3df6f66f0dfd\") " pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.306506 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.306458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchs7\" (UniqueName: \"kubernetes.io/projected/3692963f-c4d0-4368-b141-5f116a50e509-kube-api-access-lchs7\") pod \"aws-ebs-csi-driver-node-xmxcs\" (UID: \"3692963f-c4d0-4368-b141-5f116a50e509\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.379308 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.379288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:38:47.387376 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.387357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xb28h"
Sep 17 18:38:47.395071 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.395051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wj95n"
Sep 17 18:38:47.401684 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.401664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qr8gm"
Sep 17 18:38:47.412184 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.412164    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zjhmb"
Sep 17 18:38:47.417941 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.417924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtzwq"
Sep 17 18:38:47.425652 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.425635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q9dbd"
Sep 17 18:38:47.433186 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.433169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs"
Sep 17 18:38:47.437823 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.437803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:38:47.693840 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.693753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:47.693840 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.693798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:47.694054 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.693954    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:47.694054 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.693967    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:47.694054 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.694026    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:48.694006173 +0000 UTC m=+5.546710413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:47.694054 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.694048    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:38:48.694037784 +0000 UTC m=+5.546742020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:47.896301 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:47.896265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:47.896444 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.896390    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:47.896444 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.896405    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:47.896444 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.896415    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:47.896565 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:47.896473    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:48.896454453 +0000 UTC m=+5.749158677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:48.121577 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.121490    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:33:47 +0000 UTC" deadline="2028-02-23 09:37:10.30246681 +0000 UTC"
Sep 17 18:38:48.121577 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.121532    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12566h58m22.180937874s"
Sep 17 18:38:48.173115 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.173092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8be8b1_3cb1_4773_ac22_e32387848821.slice/crio-2d8c218e3edb98405c8c2c0c104857f3dd648b93d20f6fc492b1ea158e0341cd WatchSource:0}: Error finding container 2d8c218e3edb98405c8c2c0c104857f3dd648b93d20f6fc492b1ea158e0341cd: Status 404 returned error can't find the container with id 2d8c218e3edb98405c8c2c0c104857f3dd648b93d20f6fc492b1ea158e0341cd
Sep 17 18:38:48.174952 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.174819    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974224b4_09ad_4737_8b71_3df6f66f0dfd.slice/crio-f859e46001e11f19030366b76f428c745688a42890d271f3abafb5d86f98ff55 WatchSource:0}: Error finding container f859e46001e11f19030366b76f428c745688a42890d271f3abafb5d86f98ff55: Status 404 returned error can't find the container with id f859e46001e11f19030366b76f428c745688a42890d271f3abafb5d86f98ff55
Sep 17 18:38:48.178885 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.178863    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c1dd66_c309_436d_838a_f3b29ea4af63.slice/crio-88deb74be1efba8b71e9f8f7434011f11a6451b912441c732ab2378dcc0582cb WatchSource:0}: Error finding container 88deb74be1efba8b71e9f8f7434011f11a6451b912441c732ab2378dcc0582cb: Status 404 returned error can't find the container with id 88deb74be1efba8b71e9f8f7434011f11a6451b912441c732ab2378dcc0582cb
Sep 17 18:38:48.179653 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.179624    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3692963f_c4d0_4368_b141_5f116a50e509.slice/crio-aa1bdbdc68fad55166e5097aa1b7c6c53232bf8dbc81d2c2d5fa0b8a9a291d93 WatchSource:0}: Error finding container aa1bdbdc68fad55166e5097aa1b7c6c53232bf8dbc81d2c2d5fa0b8a9a291d93: Status 404 returned error can't find the container with id aa1bdbdc68fad55166e5097aa1b7c6c53232bf8dbc81d2c2d5fa0b8a9a291d93
Sep 17 18:38:48.180963 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.180938    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79480ed8_4de0_47ef_80ba_80f25ce854f1.slice/crio-52fd729cccfcc7bbc8e0b6b39cde28fdb7573a28d4303eb49aa64e7bdaf5b9be WatchSource:0}: Error finding container 52fd729cccfcc7bbc8e0b6b39cde28fdb7573a28d4303eb49aa64e7bdaf5b9be: Status 404 returned error can't find the container with id 52fd729cccfcc7bbc8e0b6b39cde28fdb7573a28d4303eb49aa64e7bdaf5b9be
Sep 17 18:38:48.182228 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.182202    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6249ccd6_6c9d_43d6_ae9d_61beefa0e564.slice/crio-880efd83387c20b100ef601cb88295f2f0a5a2c7a148c9cb4807eba4c81d8bbd WatchSource:0}: Error finding container 880efd83387c20b100ef601cb88295f2f0a5a2c7a148c9cb4807eba4c81d8bbd: Status 404 returned error can't find the container with id 880efd83387c20b100ef601cb88295f2f0a5a2c7a148c9cb4807eba4c81d8bbd
Sep 17 18:38:48.183911 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.183367    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46de969_dafc_495c_a1df_6c72a069ff78.slice/crio-ec4afb1addc4dde78e4285904bc850f254d5a916ea8a015e0d50c71e5ee697d6 WatchSource:0}: Error finding container ec4afb1addc4dde78e4285904bc850f254d5a916ea8a015e0d50c71e5ee697d6: Status 404 returned error can't find the container with id ec4afb1addc4dde78e4285904bc850f254d5a916ea8a015e0d50c71e5ee697d6
Sep 17 18:38:48.183911 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.183526    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3335f9_1553_4f57_b95a_3cd8ac5069e3.slice/crio-4abb4b5cefc5299494f578c5dfded2534219970c6934f5e07f0f53626e5f849c WatchSource:0}: Error finding container 4abb4b5cefc5299494f578c5dfded2534219970c6934f5e07f0f53626e5f849c: Status 404 returned error can't find the container with id 4abb4b5cefc5299494f578c5dfded2534219970c6934f5e07f0f53626e5f849c
Sep 17 18:38:48.185094 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:38:48.184909    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8dd36c_edaa_439a_a035_815b938708bd.slice/crio-c635f23755b59d2778094929feb68686e7e76f061c236554e3bc66624905b790 WatchSource:0}: Error finding container c635f23755b59d2778094929feb68686e7e76f061c236554e3bc66624905b790: Status 404 returned error can't find the container with id c635f23755b59d2778094929feb68686e7e76f061c236554e3bc66624905b790
Sep 17 18:38:48.193429 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.193407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:48.193546 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.193526    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:48.193619 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.193564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:48.193667 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.193631    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:48.202214 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.202191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs" event={"ID":"3692963f-c4d0-4368-b141-5f116a50e509","Type":"ContainerStarted","Data":"aa1bdbdc68fad55166e5097aa1b7c6c53232bf8dbc81d2c2d5fa0b8a9a291d93"}
Sep 17 18:38:48.203417 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.203392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"88deb74be1efba8b71e9f8f7434011f11a6451b912441c732ab2378dcc0582cb"}
Sep 17 18:38:48.204599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.204577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qr8gm" event={"ID":"974224b4-09ad-4737-8b71-3df6f66f0dfd","Type":"ContainerStarted","Data":"f859e46001e11f19030366b76f428c745688a42890d271f3abafb5d86f98ff55"}
Sep 17 18:38:48.205795 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.205762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerStarted","Data":"ec4afb1addc4dde78e4285904bc850f254d5a916ea8a015e0d50c71e5ee697d6"}
Sep 17 18:38:48.206819 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.206796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q9dbd" event={"ID":"79480ed8-4de0-47ef-80ba-80f25ce854f1","Type":"ContainerStarted","Data":"52fd729cccfcc7bbc8e0b6b39cde28fdb7573a28d4303eb49aa64e7bdaf5b9be"}
Sep 17 18:38:48.207745 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.207724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lkmvt" event={"ID":"2e8be8b1-3cb1-4773-ac22-e32387848821","Type":"ContainerStarted","Data":"2d8c218e3edb98405c8c2c0c104857f3dd648b93d20f6fc492b1ea158e0341cd"}
Sep 17 18:38:48.210955 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.210931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xb28h" event={"ID":"6c8dd36c-edaa-439a-a035-815b938708bd","Type":"ContainerStarted","Data":"c635f23755b59d2778094929feb68686e7e76f061c236554e3bc66624905b790"}
Sep 17 18:38:48.213101 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.212970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtzwq" event={"ID":"fc3335f9-1553-4f57-b95a-3cd8ac5069e3","Type":"ContainerStarted","Data":"4abb4b5cefc5299494f578c5dfded2534219970c6934f5e07f0f53626e5f849c"}
Sep 17 18:38:48.218260 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.218242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zjhmb" event={"ID":"6249ccd6-6c9d-43d6-ae9d-61beefa0e564","Type":"ContainerStarted","Data":"880efd83387c20b100ef601cb88295f2f0a5a2c7a148c9cb4807eba4c81d8bbd"}
Sep 17 18:38:48.701501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.701426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:48.701501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.701492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:48.701709 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.701620    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:48.701709 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.701682    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:50.701663654 +0000 UTC m=+7.554367879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:48.702076 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.702056    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:48.702162 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.702111    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:38:50.70209667 +0000 UTC m=+7.554800907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:48.903084 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:48.903051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:48.903261 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.903208    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:48.903261 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.903229    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:48.903261 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.903241    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:48.903411 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:48.903294    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:50.903275596 +0000 UTC m=+7.755979846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:49.122721 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.122637    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:33:47 +0000 UTC" deadline="2028-03-14 12:07:50.593745333 +0000 UTC"
Sep 17 18:38:49.122721 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.122673    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13049h29m1.471076448s"
Sep 17 18:38:49.196210 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.196180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:49.196346 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:49.196298    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:38:49.233370 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.233333    2566 generic.go:356] "Generic (PLEG): container finished" podID="a47bb08de40580886c0a61ec0484439d" containerID="8e805abebd4fb0f361017637c6667bd44279df2141a99252bc033fdfb13a2db1" exitCode=0
Sep 17 18:38:49.233965 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.233940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal" event={"ID":"a47bb08de40580886c0a61ec0484439d","Type":"ContainerDied","Data":"8e805abebd4fb0f361017637c6667bd44279df2141a99252bc033fdfb13a2db1"}
Sep 17 18:38:49.243695 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.243667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal" event={"ID":"feb5980f217da1fed6de338fd44f47c5","Type":"ContainerStarted","Data":"258a076110413af468d571eab5f7473c81fc58077f35979cce666230896b359e"}
Sep 17 18:38:49.271230 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:49.271182    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-86.ec2.internal" podStartSLOduration=3.271163855 podStartE2EDuration="3.271163855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:49.270958601 +0000 UTC m=+6.123662849" watchObservedRunningTime="2026-09-17 18:38:49.271163855 +0000 UTC m=+6.123868100"
Sep 17 18:38:50.193395 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:50.193364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:50.193890 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.193498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:50.193977 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:50.193892    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:50.194029 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.193998    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:50.266313 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:50.265876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal" event={"ID":"a47bb08de40580886c0a61ec0484439d","Type":"ContainerStarted","Data":"42b301a0bc95953b784cae598eb446b0fac4f3e426513d57d2c26a7396f65333"}
Sep 17 18:38:50.718268 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:50.718232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:50.718433 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:50.718320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:50.718433 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.718393    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:50.718546 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.718441    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:50.718546 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.718470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:54.718449227 +0000 UTC m=+11.571153464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:50.718546 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.718493    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:38:54.71847743 +0000 UTC m=+11.571181666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:50.920734 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:50.920695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:50.920936 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.920898    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:50.920936 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.920923    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:50.920936 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.920936    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:50.921088 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:50.920992    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:54.920974048 +0000 UTC m=+11.773678292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:51.193995 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:51.193927    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:51.194429 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:51.194062    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:38:52.193490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:52.193279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:52.193490 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:52.193405    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:52.193942 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:52.193752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:52.193942 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:52.193880    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:53.193952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:53.193540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:53.193952 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:53.193653    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:38:54.193883 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:54.193844    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:54.194071 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:54.193844    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:54.194071 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.193997    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:54.194530 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.194094    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:54.753421 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:54.753388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:54.753612 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:54.753448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:54.753612 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.753548    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:54.753612 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.753556    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:54.753766 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.753613    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:39:02.753593113 +0000 UTC m=+19.606297339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:54.753766 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.753633    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:02.753623378 +0000 UTC m=+19.606327606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:54.955313 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:54.955280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:54.955496 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.955461    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:54.955496 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.955491    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:54.955617 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.955505    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:54.955617 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:54.955564    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:02.955545753 +0000 UTC m=+19.808249990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:55.193792 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:55.193697    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:55.194975 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:55.194761    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:38:56.193408 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:56.193377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:56.193590 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:56.193390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:56.193590 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:56.193502    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:56.193661 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:56.193604    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:57.193133 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:57.193101    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:57.193565 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:57.193237    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:38:58.193802 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:58.193767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:38:58.194312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:58.193767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:38:58.194312 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:58.193897    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:38:58.194312 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:58.193967    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:38:59.193912 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:38:59.193880    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:38:59.194293 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:38:59.194004    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:00.193734 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:00.193700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:00.193928 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:00.193810    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:00.193928 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:00.193868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:00.194301 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:00.193990    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:01.193481 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:01.193451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:01.193688 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:01.193563    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:02.193161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:02.193124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:02.193684 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:02.193159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:02.193684 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:02.193254    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:02.193684 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:02.193401    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:02.820846 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:02.820791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:02.821038 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:02.820881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:02.821038 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:02.820950    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:02.821038 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:02.820995    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:02.821038 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:02.821022    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:39:18.821002113 +0000 UTC m=+35.673706340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:02.821038 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:02.821040    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:18.821031322 +0000 UTC m=+35.673735548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:03.022259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:03.022225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:03.022440 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:03.022367    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:03.022440 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:03.022384    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:03.022440 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:03.022393    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:03.022621 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:03.022444    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:19.022431664 +0000 UTC m=+35.875135886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:03.193904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:03.193814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:03.194387 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:03.193972    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:04.193490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:04.193452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:04.193683 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:04.193577    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:04.193683 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:04.193639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:04.193819 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:04.193781    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:05.194433 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:05.194403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:05.194769 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:05.194520    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:06.194023 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.193856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:06.194130 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.193954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:06.194194 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:06.194133    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:06.194252 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:06.194213    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:06.300752 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.300689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtzwq" event={"ID":"fc3335f9-1553-4f57-b95a-3cd8ac5069e3","Type":"ContainerStarted","Data":"1c7f40f20ca0bdedaa4c94d5f516f1db1e80efaf053443d288d13945458014cb"}
Sep 17 18:39:06.302517 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.302489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs" event={"ID":"3692963f-c4d0-4368-b141-5f116a50e509","Type":"ContainerStarted","Data":"707bbcb411d7482b05b24847077219aabcf71fd0cfb9547f2170f11f31ef634d"}
Sep 17 18:39:06.306003 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.305981    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:39:06.306369 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306348    2566 generic.go:356] "Generic (PLEG): container finished" podID="b0c1dd66-c309-436d-838a-f3b29ea4af63" containerID="392985e29a6ef0e267984b5984617adba443af1f3fa6316d0aebf7e1e37dff50" exitCode=1
Sep 17 18:39:06.306450 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"565f97c9536ff41bb488cb758c84f2ddfb6ef7855b6ad2ef0ac267d75ad06673"}
Sep 17 18:39:06.306518 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"395c0f5a6ad12c054caab6d4a179915c60784b2a1265cd794491371bac66c866"}
Sep 17 18:39:06.306576 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"0b24c1629da8ee52ecdd8899dbc5ef39bb03c64d93fdf3d4e7f15e4083bcc000"}
Sep 17 18:39:06.306576 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"e6c43f557dc8cfa56df2fee403182e353c37db0e4f4b3604b31fb8833ce14ede"}
Sep 17 18:39:06.306674 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerDied","Data":"392985e29a6ef0e267984b5984617adba443af1f3fa6316d0aebf7e1e37dff50"}
Sep 17 18:39:06.306674 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.306642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"d2c532032f684daccde8fb8e0f5ab2a9e452c547929b2d6f1a81ff59b748215f"}
Sep 17 18:39:06.307998 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.307973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qr8gm" event={"ID":"974224b4-09ad-4737-8b71-3df6f66f0dfd","Type":"ContainerStarted","Data":"9978d19d0ab226080c0c89dd75a024662fd020d693c4d1487fb7addb8985025d"}
Sep 17 18:39:06.309603 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.309577    2566 generic.go:356] "Generic (PLEG): container finished" podID="e46de969-dafc-495c-a1df-6c72a069ff78" containerID="7b56acb9f629a4401907901f9626f098754c618dcbc311aa1cf9e2265c2b4301" exitCode=0
Sep 17 18:39:06.309707 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.309658    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerDied","Data":"7b56acb9f629a4401907901f9626f098754c618dcbc311aa1cf9e2265c2b4301"}
Sep 17 18:39:06.311468 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.311445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q9dbd" event={"ID":"79480ed8-4de0-47ef-80ba-80f25ce854f1","Type":"ContainerStarted","Data":"208d0bbb5e99cc8e7e62b7a57b3bb573155f1029fc7de8733ff3e0679b67d115"}
Sep 17 18:39:06.312873 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.312852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lkmvt" event={"ID":"2e8be8b1-3cb1-4773-ac22-e32387848821","Type":"ContainerStarted","Data":"a8925826e802354f53cf7bcec86d723073905eb0f35715f6c46f82f3253417f8"}
Sep 17 18:39:06.314161 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.314143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xb28h" event={"ID":"6c8dd36c-edaa-439a-a035-815b938708bd","Type":"ContainerStarted","Data":"285e6add49b851c1d06cf6ee617d6332be1e8b9dcecfdb583b6cf5a24691c058"}
Sep 17 18:39:06.319452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.319419    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-86.ec2.internal" podStartSLOduration=20.319408701 podStartE2EDuration="20.319408701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:50.286391836 +0000 UTC m=+7.139096082" watchObservedRunningTime="2026-09-17 18:39:06.319408701 +0000 UTC m=+23.172112944"
Sep 17 18:39:06.334752 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.334718    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qtzwq" podStartSLOduration=3.872944033 podStartE2EDuration="21.334707901s" totalImagesPullingTime="17.461763868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:06.319038878 +0000 UTC m=+23.171743134" watchObservedRunningTime="2026-09-17 18:39:06.334707901 +0000 UTC m=+23.187412145"
Sep 17 18:39:06.335058 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.335037    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lkmvt" podStartSLOduration=8.15472967 podStartE2EDuration="21.335031375s" totalImagesPullingTime="13.180301705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:06.334843482 +0000 UTC m=+23.187547704" watchObservedRunningTime="2026-09-17 18:39:06.335031375 +0000 UTC m=+23.187735612"
Sep 17 18:39:06.350496 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.350465    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xb28h" podStartSLOduration=4.2488885849999996 podStartE2EDuration="21.350447444s" totalImagesPullingTime="17.101558859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:06.350293102 +0000 UTC m=+23.202997344" watchObservedRunningTime="2026-09-17 18:39:06.350447444 +0000 UTC m=+23.203151687"
Sep 17 18:39:06.411121 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.411081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q9dbd" podStartSLOduration=4.305522189 podStartE2EDuration="21.411065301s" totalImagesPullingTime="17.105543112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:06.410749003 +0000 UTC m=+23.263453249" watchObservedRunningTime="2026-09-17 18:39:06.411065301 +0000 UTC m=+23.263769547"
Sep 17 18:39:06.411213 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.411152    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qr8gm" podStartSLOduration=4.30177419 podStartE2EDuration="21.411146682s" totalImagesPullingTime="17.109372492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:06.389852225 +0000 UTC m=+23.242556464" watchObservedRunningTime="2026-09-17 18:39:06.411146682 +0000 UTC m=+23.263850925"
Sep 17 18:39:06.560193 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.560118    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lkmvt_2e8be8b1-3cb1-4773-ac22-e32387848821/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:06.688165 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:06.688140    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lkmvt_2e8be8b1-3cb1-4773-ac22-e32387848821/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:07.131869 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.131823    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:39:07.168667 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.168572    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:39:07.131862197Z","UUID":"882aa915-c12a-49c8-9d75-8e17a755d2b0","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:39:07.170051 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.170027    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:39:07.170158 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.170060    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:39:07.193154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.193132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:07.193260 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:07.193232    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:07.318459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.318381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zjhmb" event={"ID":"6249ccd6-6c9d-43d6-ae9d-61beefa0e564","Type":"ContainerStarted","Data":"a00332c8b4c3ce58b215a1816f3132cd68e80675d50e51fb065da93b9720a49c"}
Sep 17 18:39:07.320546 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.320517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs" event={"ID":"3692963f-c4d0-4368-b141-5f116a50e509","Type":"ContainerStarted","Data":"e002417fe4ad700fdacead3143bcab6a1c1c4422ebe62a01cb40d17b7f327133"}
Sep 17 18:39:07.334468 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.334425    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zjhmb" podStartSLOduration=5.232584293 podStartE2EDuration="22.334411475s" totalImagesPullingTime="17.101827182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:07.334129556 +0000 UTC m=+24.186833810" watchObservedRunningTime="2026-09-17 18:39:07.334411475 +0000 UTC m=+24.187115718"
Sep 17 18:39:07.465561 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.465532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lkmvt_2e8be8b1-3cb1-4773-ac22-e32387848821/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:07.530530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.530504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lkmvt_2e8be8b1-3cb1-4773-ac22-e32387848821/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:07.584400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:07.584339    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lkmvt_2e8be8b1-3cb1-4773-ac22-e32387848821/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:08.193981 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:08.193948    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:08.193981 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:08.193964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:08.194211 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:08.194065    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:08.194211 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:08.194174    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:09.193952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:09.193748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:09.194348 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:09.194032    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:09.327558 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:09.327526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs" event={"ID":"3692963f-c4d0-4368-b141-5f116a50e509","Type":"ContainerStarted","Data":"1e7cd3f2f2ac84a6e837159a751b92d72d983b73ba2e54a2ae124edca2ac4584"}
Sep 17 18:39:09.331127 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:09.331106    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:39:09.331530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:09.331495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"bcbfc3a0ebebb994c4fb3dad7c2ef1b758d109c2499d7867b3a7d586fe48d496"}
Sep 17 18:39:09.346157 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:09.346110    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmxcs" podStartSLOduration=4.718357096 podStartE2EDuration="24.346096112s" totalImagesPullingTime="19.627739016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:09.34568772 +0000 UTC m=+26.198391965" watchObservedRunningTime="2026-09-17 18:39:09.346096112 +0000 UTC m=+26.198800356"
Sep 17 18:39:10.193649 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:10.193618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:10.193847 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:10.193619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:10.193847 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:10.193751    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:10.193847 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:10.193790    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:10.716485 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:10.716453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:39:10.717177 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:10.717150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:39:11.193946 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:11.193915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:11.194104 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:11.194037    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:11.336893 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:11.336866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:39:11.337397 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:11.337377    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lkmvt"
Sep 17 18:39:12.193904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.193713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:12.194425 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.193786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:12.194425 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:12.193983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:12.194425 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:12.194056    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:12.342288 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.342269    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:39:12.342648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.342627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"2810404b63dab8db957484cc281d199dc2fb9a4f6241ddbdee1192d57e56b973"}
Sep 17 18:39:12.342952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.342928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:39:12.343052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.342994    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:39:12.343128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.343113    2566 scope.go:117] "RemoveContainer" containerID="392985e29a6ef0e267984b5984617adba443af1f3fa6316d0aebf7e1e37dff50"
Sep 17 18:39:12.344741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.344712    2566 generic.go:356] "Generic (PLEG): container finished" podID="e46de969-dafc-495c-a1df-6c72a069ff78" containerID="2ccd3d1955361faae73e83454fa3c22524ba1e691997ba13b02eeef0512e0906" exitCode=0
Sep 17 18:39:12.344901 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.344763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerDied","Data":"2ccd3d1955361faae73e83454fa3c22524ba1e691997ba13b02eeef0512e0906"}
Sep 17 18:39:12.358399 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:12.358380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:39:13.193215 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.193188    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:13.193403 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:13.193311    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:13.350145 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.350121    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:39:13.350491 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.350461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" event={"ID":"b0c1dd66-c309-436d-838a-f3b29ea4af63","Type":"ContainerStarted","Data":"1b5e41cd1c0021d7b599d612094d55d14286ba1ac22aaed7ee74f3814f2303ee"}
Sep 17 18:39:13.350688 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.350672    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:39:13.365681 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.365662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:39:13.382776 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.382733    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls" podStartSLOduration=11.220228343 podStartE2EDuration="28.382720948s" totalImagesPullingTime="17.162492605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:13.381336268 +0000 UTC m=+30.234040513" watchObservedRunningTime="2026-09-17 18:39:13.382720948 +0000 UTC m=+30.235425191"
Sep 17 18:39:13.506742 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.506681    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qfzqm"]
Sep 17 18:39:13.506915 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.506788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:13.506982 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:13.506904    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:13.508628 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.508583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bvtmd"]
Sep 17 18:39:13.508727 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.508686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:13.508783 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:13.508766    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:13.516164 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.516140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9gtzs"]
Sep 17 18:39:13.516274 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:13.516219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:13.516339 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:13.516312    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:14.354724 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:14.354693    2566 generic.go:356] "Generic (PLEG): container finished" podID="e46de969-dafc-495c-a1df-6c72a069ff78" containerID="ed7708d45e53dce9273be9de4c18a9a47e2379ab99ea2966401bd36279a0703c" exitCode=0
Sep 17 18:39:14.355091 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:14.354776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerDied","Data":"ed7708d45e53dce9273be9de4c18a9a47e2379ab99ea2966401bd36279a0703c"}
Sep 17 18:39:15.195782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:15.195580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:15.196004 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:15.195966    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:15.196466 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:15.196448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:15.196637 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:15.196617    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:15.196693 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:15.196677    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:15.196857 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:15.196820    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:15.360082 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:15.360056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerStarted","Data":"3fd535882c5b62d5aa85d97db75107ec2069c0d9a285328863ba37c928b5ccbc"}
Sep 17 18:39:16.364794 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:16.364764    2566 generic.go:356] "Generic (PLEG): container finished" podID="e46de969-dafc-495c-a1df-6c72a069ff78" containerID="3fd535882c5b62d5aa85d97db75107ec2069c0d9a285328863ba37c928b5ccbc" exitCode=0
Sep 17 18:39:16.365137 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:16.364808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerDied","Data":"3fd535882c5b62d5aa85d97db75107ec2069c0d9a285328863ba37c928b5ccbc"}
Sep 17 18:39:17.193361 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:17.193334    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:17.193515 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:17.193334    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:17.193515 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:17.193426    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qfzqm" podUID="4e1e9522-f9b9-4675-a969-084168f1bb41"
Sep 17 18:39:17.193515 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:17.193444    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:17.193639 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:17.193525    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvtmd" podUID="853ece52-4596-4c55-83af-da02fc283b14"
Sep 17 18:39:17.193639 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:17.193591    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9gtzs" podUID="1e4b8718-ff20-4f8b-bfee-853b19603fdc"
Sep 17 18:39:18.500696 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.500617    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-86.ec2.internal" event="NodeReady"
Sep 17 18:39:18.501169 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.500722    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:39:18.545522 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.545494    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-khlll"]
Sep 17 18:39:18.565977 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.565944    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b78d2"]
Sep 17 18:39:18.566224 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.566084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.568580 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.568532    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:39:18.568714 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.568618    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zttzk\""
Sep 17 18:39:18.568714 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.568632    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:39:18.590545 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.590525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khlll"]
Sep 17 18:39:18.590650 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.590558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b78d2"]
Sep 17 18:39:18.590713 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.590653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.593104 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.592938    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:39:18.593104 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.593083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jpf78\""
Sep 17 18:39:18.593337 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.593109    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:39:18.593337 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.593135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:39:18.738940 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.738896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-kube-api-access-sj7pf\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.738940 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.738934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83139208-88be-4b5e-84ea-365d834265ec-cert\") pod \"ingress-canary-b78d2\" (UID: \"83139208-88be-4b5e-84ea-365d834265ec\") " pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.739208 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.739009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-config-volume\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.739208 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.739035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.739208 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.739119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-tmp-dir\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.739208 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.739181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsgk\" (UniqueName: \"kubernetes.io/projected/83139208-88be-4b5e-84ea-365d834265ec-kube-api-access-rmsgk\") pod \"ingress-canary-b78d2\" (UID: \"83139208-88be-4b5e-84ea-365d834265ec\") " pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.839669 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:18.839815 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsgk\" (UniqueName: \"kubernetes.io/projected/83139208-88be-4b5e-84ea-365d834265ec-kube-api-access-rmsgk\") pod \"ingress-canary-b78d2\" (UID: \"83139208-88be-4b5e-84ea-365d834265ec\") " pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.839815 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-kube-api-access-sj7pf\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.839815 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83139208-88be-4b5e-84ea-365d834265ec-cert\") pod \"ingress-canary-b78d2\" (UID: \"83139208-88be-4b5e-84ea-365d834265ec\") " pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.839815 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:18.839815 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-config-volume\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.839815 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:18.839810    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:18.840101 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.839825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.840101 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:18.839892    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs podName:1e4b8718-ff20-4f8b-bfee-853b19603fdc nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.839872815 +0000 UTC m=+67.692577051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs") pod "network-metrics-daemon-9gtzs" (UID: "1e4b8718-ff20-4f8b-bfee-853b19603fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:18.840101 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:18.839917    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:18.840101 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:18.839960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls podName:d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:19.339946246 +0000 UTC m=+36.192650468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls") pod "dns-default-khlll" (UID: "d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:18.840308 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:18.840111    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:18.840308 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.840159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-tmp-dir\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.840308 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:18.840170    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret podName:853ece52-4596-4c55-83af-da02fc283b14 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.840156046 +0000 UTC m=+67.692860273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret") pod "global-pull-secret-syncer-bvtmd" (UID: "853ece52-4596-4c55-83af-da02fc283b14") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:18.840413 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.840370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-tmp-dir\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.840523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.840503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-config-volume\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.845195 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.845100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83139208-88be-4b5e-84ea-365d834265ec-cert\") pod \"ingress-canary-b78d2\" (UID: \"83139208-88be-4b5e-84ea-365d834265ec\") " pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.847865 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.847843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-kube-api-access-sj7pf\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:18.848025 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.847996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsgk\" (UniqueName: \"kubernetes.io/projected/83139208-88be-4b5e-84ea-365d834265ec-kube-api-access-rmsgk\") pod \"ingress-canary-b78d2\" (UID: \"83139208-88be-4b5e-84ea-365d834265ec\") " pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:18.900786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:18.900761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b78d2"
Sep 17 18:39:19.041627 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.041467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:19.041740 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:19.041635    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:19.041740 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:19.041655    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:19.041740 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:19.041666    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6cxkl for pod openshift-network-diagnostics/network-check-target-qfzqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:19.041740 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:19.041727    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl podName:4e1e9522-f9b9-4675-a969-084168f1bb41 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:51.041712869 +0000 UTC m=+67.894417091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cxkl" (UniqueName: "kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl") pod "network-check-target-qfzqm" (UID: "4e1e9522-f9b9-4675-a969-084168f1bb41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:19.193490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.193398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:19.193490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.193415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:19.193490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.193412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:19.196365 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.196068    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:39:19.196365 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.196106    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:19.196365 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.196110    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:39:19.196365 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.196131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kfmzw\""
Sep 17 18:39:19.196637 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.196510    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gk2t2\""
Sep 17 18:39:19.196689 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.196678    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:19.230457 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.230429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b78d2"]
Sep 17 18:39:19.234604 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:19.234579    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83139208_88be_4b5e_84ea_365d834265ec.slice/crio-4ad96eca035a1dfaf0dc31febd4c70988ca5d1dcee75876966b1fca48b57453b WatchSource:0}: Error finding container 4ad96eca035a1dfaf0dc31febd4c70988ca5d1dcee75876966b1fca48b57453b: Status 404 returned error can't find the container with id 4ad96eca035a1dfaf0dc31febd4c70988ca5d1dcee75876966b1fca48b57453b
Sep 17 18:39:19.343032 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.343009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:19.343167 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:19.343150    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:19.343214 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:19.343210    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls podName:d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:20.343194006 +0000 UTC m=+37.195898241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls") pod "dns-default-khlll" (UID: "d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:19.372793 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:19.372751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b78d2" event={"ID":"83139208-88be-4b5e-84ea-365d834265ec","Type":"ContainerStarted","Data":"4ad96eca035a1dfaf0dc31febd4c70988ca5d1dcee75876966b1fca48b57453b"}
Sep 17 18:39:20.353339 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:20.353295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:20.353983 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:20.353473    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:20.353983 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:20.353537    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls podName:d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.353514762 +0000 UTC m=+39.206218984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls") pod "dns-default-khlll" (UID: "d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.367753 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:22.367685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:22.368093 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:22.367852    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.368093 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:22.367915    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls podName:d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:26.367899448 +0000 UTC m=+43.220603673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls") pod "dns-default-khlll" (UID: "d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.382523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:22.382502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b78d2" event={"ID":"83139208-88be-4b5e-84ea-365d834265ec","Type":"ContainerStarted","Data":"20293d3d4bf75bc1ead31167c192f3a09008b1f8cdb8a60caf5b9506c2e5891d"}
Sep 17 18:39:22.385218 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:22.385195    2566 generic.go:356] "Generic (PLEG): container finished" podID="e46de969-dafc-495c-a1df-6c72a069ff78" containerID="6c422631395f0fbce47fb014d9abdb05cebeff5a570fdd94199bfb27bdd45106" exitCode=0
Sep 17 18:39:22.385312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:22.385232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerDied","Data":"6c422631395f0fbce47fb014d9abdb05cebeff5a570fdd94199bfb27bdd45106"}
Sep 17 18:39:22.421369 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:22.421325    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b78d2" podStartSLOduration=1.553470041 podStartE2EDuration="4.421310812s" totalImagesPullingTime="2.867840771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:22.420969102 +0000 UTC m=+39.273673346" watchObservedRunningTime="2026-09-17 18:39:22.421310812 +0000 UTC m=+39.274015055"
Sep 17 18:39:23.390112 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:23.389929    2566 generic.go:356] "Generic (PLEG): container finished" podID="e46de969-dafc-495c-a1df-6c72a069ff78" containerID="a279814b8710578c02b1b005fe85dd83b4b963f9fc21b98dac7158ff908c0a9a" exitCode=0
Sep 17 18:39:23.390459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:23.390005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerDied","Data":"a279814b8710578c02b1b005fe85dd83b4b963f9fc21b98dac7158ff908c0a9a"}
Sep 17 18:39:24.396679 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:24.396643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wj95n" event={"ID":"e46de969-dafc-495c-a1df-6c72a069ff78","Type":"ContainerStarted","Data":"e5acd25a38de36694b226ebaa472ba76d1b01d504149411139e390d6561bb8cf"}
Sep 17 18:39:24.418933 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:24.418881    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wj95n" podStartSLOduration=8.878108983 podStartE2EDuration="39.418866227s" totalImagesPullingTime="30.540757244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:24.417493182 +0000 UTC m=+41.270197427" watchObservedRunningTime="2026-09-17 18:39:24.418866227 +0000 UTC m=+41.271570471"
Sep 17 18:39:25.558768 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.558730    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"]
Sep 17 18:39:25.561679 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.561660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.564627 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.564598    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 18:39:25.565355 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.565329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:25.565489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.565376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pt8cl\""
Sep 17 18:39:25.565489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.565386    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 18:39:25.565489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.565424    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:25.568319 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.568291    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"]
Sep 17 18:39:25.571180 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.571159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:25.573149 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.573119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"]
Sep 17 18:39:25.573390 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.573374    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:25.573479 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.573409    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 18:39:25.573760 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.573740    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:25.573881 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.573864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lk7hn\""
Sep 17 18:39:25.581603 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.581575    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"]
Sep 17 18:39:25.665908 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.665883    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"]
Sep 17 18:39:25.668579 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.668564    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8476b68445-qgzvv"]
Sep 17 18:39:25.668688 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.668673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.671207 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.671186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.671482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.671460    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:25.671642 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.671626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 18:39:25.671687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.671653    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:25.671687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.671668    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 18:39:25.671790 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.671765    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2q68k\""
Sep 17 18:39:25.673261 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.673245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:39:25.673704 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.673689    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dw5r6\""
Sep 17 18:39:25.673789 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.673724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:39:25.674861 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.674822    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:39:25.678947 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.678932    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:39:25.687129 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8476b68445-qgzvv"]
Sep 17 18:39:25.687353 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssh7\" (UniqueName: \"kubernetes.io/projected/bfe11238-6e44-48c6-8129-762bf456a5c8-kube-api-access-lssh7\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:25.687464 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:25.687464 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdt2\" (UniqueName: \"kubernetes.io/projected/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-kube-api-access-5bdt2\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.687606 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-serving-cert\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.687606 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-config\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.687783 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.687767    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"]
Sep 17 18:39:25.788636 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:25.788730 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-image-registry-private-configuration\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.788730 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-trusted-ca\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.788730 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b33c6fa3-056b-490b-991a-afde594f8f62-ca-trust-extracted\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.788826 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:25.788757    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:39:25.788826 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdt2\" (UniqueName: \"kubernetes.io/projected/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-kube-api-access-5bdt2\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.788826 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:25.788803    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls podName:bfe11238-6e44-48c6-8129-762bf456a5c8 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:26.288791233 +0000 UTC m=+43.141495456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-phwc6" (UID: "bfe11238-6e44-48c6-8129-762bf456a5c8") : secret "samples-operator-tls" not found
Sep 17 18:39:25.788951 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-serving-cert\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.788993 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-config\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.788993 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.788982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-bound-sa-token\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.789087 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.789087 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-registry-certificates\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.789087 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jprr\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-kube-api-access-2jprr\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.789087 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssh7\" (UniqueName: \"kubernetes.io/projected/bfe11238-6e44-48c6-8129-762bf456a5c8-kube-api-access-lssh7\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:25.789281 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-installation-pull-secrets\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.789281 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d81bfd12-ec20-4803-b867-dc1401885094-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.789281 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d81bfd12-ec20-4803-b867-dc1401885094-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.789281 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsc7g\" (UniqueName: \"kubernetes.io/projected/d81bfd12-ec20-4803-b867-dc1401885094-kube-api-access-qsc7g\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.789499 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.789483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-config\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.792725 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.792708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-serving-cert\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.804879 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.804855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdt2\" (UniqueName: \"kubernetes.io/projected/ab21b025-a42f-4bfb-bbe2-16ab6d0d5109-kube-api-access-5bdt2\") pod \"service-ca-operator-57cdc4b768-hkp99\" (UID: \"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.807051 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.807034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssh7\" (UniqueName: \"kubernetes.io/projected/bfe11238-6e44-48c6-8129-762bf456a5c8-kube-api-access-lssh7\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:25.871092 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.871046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"
Sep 17 18:39:25.889857 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.889816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-installation-pull-secrets\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.889949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.889879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d81bfd12-ec20-4803-b867-dc1401885094-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.889949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.889905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d81bfd12-ec20-4803-b867-dc1401885094-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.889949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.889939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsc7g\" (UniqueName: \"kubernetes.io/projected/d81bfd12-ec20-4803-b867-dc1401885094-kube-api-access-qsc7g\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.890128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-image-registry-private-configuration\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890232 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-trusted-ca\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890232 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b33c6fa3-056b-490b-991a-afde594f8f62-ca-trust-extracted\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-bound-sa-token\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-registry-certificates\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jprr\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-kube-api-access-2jprr\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d81bfd12-ec20-4803-b867-dc1401885094-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:25.890622    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:25.890637    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8476b68445-qgzvv: secret "image-registry-tls" not found
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.890662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b33c6fa3-056b-490b-991a-afde594f8f62-ca-trust-extracted\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.890904 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:25.890698    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls podName:b33c6fa3-056b-490b-991a-afde594f8f62 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:26.390679203 +0000 UTC m=+43.243383431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls") pod "image-registry-8476b68445-qgzvv" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62") : secret "image-registry-tls" not found
Sep 17 18:39:25.891446 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.891041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-registry-certificates\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.891446 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.891338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-trusted-ca\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.892657 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.892389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d81bfd12-ec20-4803-b867-dc1401885094-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.892657 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.892510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-installation-pull-secrets\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.892943 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.892927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-image-registry-private-configuration\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.899017 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.898999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsc7g\" (UniqueName: \"kubernetes.io/projected/d81bfd12-ec20-4803-b867-dc1401885094-kube-api-access-qsc7g\") pod \"kube-storage-version-migrator-operator-55889f7b47-jfxvf\" (UID: \"d81bfd12-ec20-4803-b867-dc1401885094\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:25.903188 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.903165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-bound-sa-token\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.905235 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.905212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jprr\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-kube-api-access-2jprr\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:25.977657 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:25.977628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"
Sep 17 18:39:26.145206 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.145145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99"]
Sep 17 18:39:26.148781 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:26.148756    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab21b025_a42f_4bfb_bbe2_16ab6d0d5109.slice/crio-cbaf42e7459fdea5c933980272ebb4d15c112a0fac174d77fa004330bec92cde WatchSource:0}: Error finding container cbaf42e7459fdea5c933980272ebb4d15c112a0fac174d77fa004330bec92cde: Status 404 returned error can't find the container with id cbaf42e7459fdea5c933980272ebb4d15c112a0fac174d77fa004330bec92cde
Sep 17 18:39:26.252265 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.252242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf"]
Sep 17 18:39:26.254845 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:26.254805    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81bfd12_ec20_4803_b867_dc1401885094.slice/crio-f4366ca46c6ce8c656ab3a3c0637d89552e2596cb9ec261dfa27bf86f8f893ce WatchSource:0}: Error finding container f4366ca46c6ce8c656ab3a3c0637d89552e2596cb9ec261dfa27bf86f8f893ce: Status 404 returned error can't find the container with id f4366ca46c6ce8c656ab3a3c0637d89552e2596cb9ec261dfa27bf86f8f893ce
Sep 17 18:39:26.293342 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.293319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:26.293450 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.293438    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:39:26.293497 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.293491    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls podName:bfe11238-6e44-48c6-8129-762bf456a5c8 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:27.293478231 +0000 UTC m=+44.146182453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-phwc6" (UID: "bfe11238-6e44-48c6-8129-762bf456a5c8") : secret "samples-operator-tls" not found
Sep 17 18:39:26.393979 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.393950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:26.394065 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.393994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:26.394109 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.394090    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:26.394109 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.394106    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8476b68445-qgzvv: secret "image-registry-tls" not found
Sep 17 18:39:26.394176 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.394115    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:26.394176 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.394151    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls podName:b33c6fa3-056b-490b-991a-afde594f8f62 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:27.394137593 +0000 UTC m=+44.246841820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls") pod "image-registry-8476b68445-qgzvv" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62") : secret "image-registry-tls" not found
Sep 17 18:39:26.394176 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:26.394164    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls podName:d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:34.394158179 +0000 UTC m=+51.246862401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls") pod "dns-default-khlll" (UID: "d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:26.402078 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.402015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf" event={"ID":"d81bfd12-ec20-4803-b867-dc1401885094","Type":"ContainerStarted","Data":"f4366ca46c6ce8c656ab3a3c0637d89552e2596cb9ec261dfa27bf86f8f893ce"}
Sep 17 18:39:26.403121 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:26.403100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99" event={"ID":"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109","Type":"ContainerStarted","Data":"cbaf42e7459fdea5c933980272ebb4d15c112a0fac174d77fa004330bec92cde"}
Sep 17 18:39:27.301342 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.301307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:27.301791 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:27.301476    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:39:27.301791 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:27.301563    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls podName:bfe11238-6e44-48c6-8129-762bf456a5c8 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:29.301540508 +0000 UTC m=+46.154244736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-phwc6" (UID: "bfe11238-6e44-48c6-8129-762bf456a5c8") : secret "samples-operator-tls" not found
Sep 17 18:39:27.402519 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.402488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:27.402656 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:27.402611    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:27.402656 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:27.402623    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8476b68445-qgzvv: secret "image-registry-tls" not found
Sep 17 18:39:27.402746 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:27.402673    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls podName:b33c6fa3-056b-490b-991a-afde594f8f62 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:29.402656922 +0000 UTC m=+46.255361144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls") pod "image-registry-8476b68445-qgzvv" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62") : secret "image-registry-tls" not found
Sep 17 18:39:27.930065 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.930033    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"]
Sep 17 18:39:27.933771 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.933748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:27.936494 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.936474    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 18:39:27.936581 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.936497    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nr5s7\""
Sep 17 18:39:27.936581 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.936530    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 18:39:27.943420 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:27.943398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"]
Sep 17 18:39:28.109535 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:28.109504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:28.109682 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:28.109538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:28.210575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:28.210493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:28.210575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:28.210531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:28.210842 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:28.210745    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:39:28.210842 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:28.210841    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert podName:3d8cb0ee-0042-48b9-91f4-3f29fee8230e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:28.71080591 +0000 UTC m=+45.563510145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jcfxc" (UID: "3d8cb0ee-0042-48b9-91f4-3f29fee8230e") : secret "networking-console-plugin-cert" not found
Sep 17 18:39:28.211207 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:28.211173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:28.714718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:28.714682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:28.715089 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:28.714816    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:39:28.715089 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:28.714899    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert podName:3d8cb0ee-0042-48b9-91f4-3f29fee8230e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:29.714880161 +0000 UTC m=+46.567584396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jcfxc" (UID: "3d8cb0ee-0042-48b9-91f4-3f29fee8230e") : secret "networking-console-plugin-cert" not found
Sep 17 18:39:29.318698 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.318663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:29.318818 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.318803    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:39:29.318893 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.318870    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls podName:bfe11238-6e44-48c6-8129-762bf456a5c8 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:33.318854062 +0000 UTC m=+50.171558292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-phwc6" (UID: "bfe11238-6e44-48c6-8129-762bf456a5c8") : secret "samples-operator-tls" not found
Sep 17 18:39:29.411736 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.411703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf" event={"ID":"d81bfd12-ec20-4803-b867-dc1401885094","Type":"ContainerStarted","Data":"f544a42418fe633e594110c06a5a4391eddcfaf410439e839f9327048b8cc839"}
Sep 17 18:39:29.413295 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.413263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99" event={"ID":"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109","Type":"ContainerStarted","Data":"c77feb24540513cff4c91a996b14a1c4848d88a7643e536c4c87d3f462e73886"}
Sep 17 18:39:29.419668 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.419645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:29.419778 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.419764    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:29.420014 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.419780    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8476b68445-qgzvv: secret "image-registry-tls" not found
Sep 17 18:39:29.420014 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.419825    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls podName:b33c6fa3-056b-490b-991a-afde594f8f62 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:33.419809883 +0000 UTC m=+50.272514119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls") pod "image-registry-8476b68445-qgzvv" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62") : secret "image-registry-tls" not found
Sep 17 18:39:29.429921 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.429877    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf" podStartSLOduration=2.112665599 podStartE2EDuration="4.429861819s" totalImagesPullingTime="2.31719622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:29.428587366 +0000 UTC m=+46.281291611" watchObservedRunningTime="2026-09-17 18:39:29.429861819 +0000 UTC m=+46.282566066"
Sep 17 18:39:29.448743 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.448624    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99" podStartSLOduration=1.3544928999999999 podStartE2EDuration="4.448610232s" totalImagesPullingTime="3.094117332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:29.448225109 +0000 UTC m=+46.300929354" watchObservedRunningTime="2026-09-17 18:39:29.448610232 +0000 UTC m=+46.301314479"
Sep 17 18:39:29.721217 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.721189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:29.721636 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.721362    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:39:29.721636 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:29.721444    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert podName:3d8cb0ee-0042-48b9-91f4-3f29fee8230e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:31.721424117 +0000 UTC m=+48.574128358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jcfxc" (UID: "3d8cb0ee-0042-48b9-91f4-3f29fee8230e") : secret "networking-console-plugin-cert" not found
Sep 17 18:39:29.829417 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.829390    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"]
Sep 17 18:39:29.832772 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.832753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"
Sep 17 18:39:29.835256 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.835213    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:29.835371 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.835310    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 18:39:29.835846 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.835815    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-24n4q\""
Sep 17 18:39:29.842169 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.842146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"]
Sep 17 18:39:29.923364 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:29.923337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qthg\" (UniqueName: \"kubernetes.io/projected/84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5-kube-api-access-9qthg\") pod \"migrator-76794cbd6c-qs6vs\" (UID: \"84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"
Sep 17 18:39:30.024084 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.024035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qthg\" (UniqueName: \"kubernetes.io/projected/84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5-kube-api-access-9qthg\") pod \"migrator-76794cbd6c-qs6vs\" (UID: \"84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"
Sep 17 18:39:30.032523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.032499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qthg\" (UniqueName: \"kubernetes.io/projected/84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5-kube-api-access-9qthg\") pod \"migrator-76794cbd6c-qs6vs\" (UID: \"84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"
Sep 17 18:39:30.143374 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.143348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"
Sep 17 18:39:30.435420 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.435393    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs"]
Sep 17 18:39:30.438599 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:30.438573    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a16572_8eb4_4eba_8a1a_70a6f7b9b5c5.slice/crio-04502e9e3c312242d0a1a11f839d859c54efd1e53387af7f973c3112a6127e2d WatchSource:0}: Error finding container 04502e9e3c312242d0a1a11f839d859c54efd1e53387af7f973c3112a6127e2d: Status 404 returned error can't find the container with id 04502e9e3c312242d0a1a11f839d859c54efd1e53387af7f973c3112a6127e2d
Sep 17 18:39:30.571109 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.571085    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nfkgq"]
Sep 17 18:39:30.610512 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.610477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nfkgq"]
Sep 17 18:39:30.610614 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.610599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.613226 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.613204    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:39:30.613299 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.613257    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:39:30.613925 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.613906    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:39:30.614027 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.614009    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:39:30.614281 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.614265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtkmh\""
Sep 17 18:39:30.629693 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.629481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c3a10-a9d4-40a5-b36a-d644cb3b548a-data-volume\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.632147 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.629719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvt7\" (UniqueName: \"kubernetes.io/projected/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-api-access-njvt7\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.632147 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.630005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/198c3a10-a9d4-40a5-b36a-d644cb3b548a-crio-socket\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.632147 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.630110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.632147 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.630233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732511 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732511 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c3a10-a9d4-40a5-b36a-d644cb3b548a-data-volume\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvt7\" (UniqueName: \"kubernetes.io/projected/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-api-access-njvt7\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/198c3a10-a9d4-40a5-b36a-d644cb3b548a-crio-socket\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:30.732604    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:30.732669    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:31.232648577 +0000 UTC m=+48.085352799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:30.732812    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/198c3a10-a9d4-40a5-b36a-d644cb3b548a-crio-socket\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.732868 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.732819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c3a10-a9d4-40a5-b36a-d644cb3b548a-data-volume\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:30.733118 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:30.732880    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:31.232867971 +0000 UTC m=+48.085572195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:30.740561 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:30.740541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvt7\" (UniqueName: \"kubernetes.io/projected/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-api-access-njvt7\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:31.237581 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:31.237551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:31.237741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:31.237621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:31.237741 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:31.237689    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:31.237877 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:31.237766    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:31.237877 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:31.237771    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:32.237751232 +0000 UTC m=+49.090455470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:31.237877 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:31.237845    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:32.237812923 +0000 UTC m=+49.090517148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:31.421007 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:31.420971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs" event={"ID":"84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5","Type":"ContainerStarted","Data":"04502e9e3c312242d0a1a11f839d859c54efd1e53387af7f973c3112a6127e2d"}
Sep 17 18:39:31.740428 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:31.740381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:31.740814 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:31.740566    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:39:31.740814 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:31.740634    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert podName:3d8cb0ee-0042-48b9-91f4-3f29fee8230e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:35.740618828 +0000 UTC m=+52.593323069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jcfxc" (UID: "3d8cb0ee-0042-48b9-91f4-3f29fee8230e") : secret "networking-console-plugin-cert" not found
Sep 17 18:39:32.244633 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:32.244606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:32.244732 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:32.244657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:32.244732 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:32.244709    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:32.244804 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:32.244753    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:32.244804 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:32.244762    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:34.2447474 +0000 UTC m=+51.097451639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:32.244804 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:32.244794    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:34.244787643 +0000 UTC m=+51.097491866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:32.425684 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:32.425661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs" event={"ID":"84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5","Type":"ContainerStarted","Data":"d4351fd0ff8f68a1d670a9d2ab9f7454c7d5cee9be77a771e49ce37f89b98755"}
Sep 17 18:39:32.425792 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:32.425717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs" event={"ID":"84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5","Type":"ContainerStarted","Data":"71a00c4fda5447aa710546da88078bb191c95d9b46302c6df209eb324236f688"}
Sep 17 18:39:33.352766 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:33.352731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:33.353268 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:33.352874    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:39:33.353268 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:33.352928    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls podName:bfe11238-6e44-48c6-8129-762bf456a5c8 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:41.352915112 +0000 UTC m=+58.205619335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-phwc6" (UID: "bfe11238-6e44-48c6-8129-762bf456a5c8") : secret "samples-operator-tls" not found
Sep 17 18:39:33.453032 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:33.453005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:33.453193 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:33.453147    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:33.453193 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:33.453164    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8476b68445-qgzvv: secret "image-registry-tls" not found
Sep 17 18:39:33.453263 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:33.453226    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls podName:b33c6fa3-056b-490b-991a-afde594f8f62 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:41.453207585 +0000 UTC m=+58.305911810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls") pod "image-registry-8476b68445-qgzvv" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62") : secret "image-registry-tls" not found
Sep 17 18:39:33.525645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:33.525623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q9dbd_79480ed8-4de0-47ef-80ba-80f25ce854f1/dns-node-resolver/0.log"
Sep 17 18:39:34.258904 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:34.258872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:34.259066 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:34.258925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:34.259066 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:34.259026    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:34.259137 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:34.259078    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:34.259137 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:34.259102    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.259085153 +0000 UTC m=+55.111789411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:34.259137 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:34.259125    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:38.259112369 +0000 UTC m=+55.111816679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:34.317167 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:34.317143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qr8gm_974224b4-09ad-4737-8b71-3df6f66f0dfd/node-ca/0.log"
Sep 17 18:39:34.460024 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:34.459999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:34.460329 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:34.460167    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:34.460329 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:34.460224    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls podName:d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.460210751 +0000 UTC m=+67.312915007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls") pod "dns-default-khlll" (UID: "d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:34.917261 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:34.917228    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b78d2_83139208-88be-4b5e-84ea-365d834265ec/serve-healthcheck-canary/0.log"
Sep 17 18:39:35.524598 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:35.524573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qs6vs_84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5/migrator/0.log"
Sep 17 18:39:35.719786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:35.719758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qs6vs_84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5/graceful-termination/0.log"
Sep 17 18:39:35.770082 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:35.770054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:35.770212 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:35.770197    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:39:35.770273 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:35.770264    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert podName:3d8cb0ee-0042-48b9-91f4-3f29fee8230e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:43.770248978 +0000 UTC m=+60.622953205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jcfxc" (UID: "3d8cb0ee-0042-48b9-91f4-3f29fee8230e") : secret "networking-console-plugin-cert" not found
Sep 17 18:39:35.918251 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:35.918226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jfxvf_d81bfd12-ec20-4803-b867-dc1401885094/kube-storage-version-migrator-operator/0.log"
Sep 17 18:39:38.287511 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:38.287470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:38.287945 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:38.287538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:38.287945 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:38.287618    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:39:38.287945 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:38.287662    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:38.287945 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:38.287688    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:46.287667755 +0000 UTC m=+63.140371999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : configmap "kube-rbac-proxy" not found
Sep 17 18:39:38.287945 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:38.287744    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:39:46.287737227 +0000 UTC m=+63.140441448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:41.411103 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.411064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:41.413582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.413556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfe11238-6e44-48c6-8129-762bf456a5c8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-phwc6\" (UID: \"bfe11238-6e44-48c6-8129-762bf456a5c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:41.480682 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.480653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"
Sep 17 18:39:41.512135 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.512108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:41.514432 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.514413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"image-registry-8476b68445-qgzvv\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") " pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:41.583173 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.583140    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:41.789460 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.789379    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qs6vs" podStartSLOduration=11.043608756 podStartE2EDuration="12.789360576s" totalImagesPullingTime="1.74575182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:32.446166639 +0000 UTC m=+49.298870885" watchObservedRunningTime="2026-09-17 18:39:41.789360576 +0000 UTC m=+58.642064819"
Sep 17 18:39:41.790177 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.790160    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6"]
Sep 17 18:39:41.890079 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:41.890056    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8476b68445-qgzvv"]
Sep 17 18:39:41.892798 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:41.892773    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33c6fa3_056b_490b_991a_afde594f8f62.slice/crio-f7ec866ef2f1c4858b24a5de9de3e32e5d39e0d648d767bc7eb1326c66e39a7c WatchSource:0}: Error finding container f7ec866ef2f1c4858b24a5de9de3e32e5d39e0d648d767bc7eb1326c66e39a7c: Status 404 returned error can't find the container with id f7ec866ef2f1c4858b24a5de9de3e32e5d39e0d648d767bc7eb1326c66e39a7c
Sep 17 18:39:42.458095 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:42.458057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" event={"ID":"b33c6fa3-056b-490b-991a-afde594f8f62","Type":"ContainerStarted","Data":"66a4f0be4f65f1a3e89425fdd116ebecc272c05f1f3f7e9773041467d17b2afd"}
Sep 17 18:39:42.458521 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:42.458110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" event={"ID":"b33c6fa3-056b-490b-991a-afde594f8f62","Type":"ContainerStarted","Data":"f7ec866ef2f1c4858b24a5de9de3e32e5d39e0d648d767bc7eb1326c66e39a7c"}
Sep 17 18:39:42.458521 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:42.458176    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:39:42.459381 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:42.459355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6" event={"ID":"bfe11238-6e44-48c6-8129-762bf456a5c8","Type":"ContainerStarted","Data":"b6c7b6c81b1d951186d365d21365e97e60c4615155ff5ae71dd384d48e40f468"}
Sep 17 18:39:42.488553 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:42.488515    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" podStartSLOduration=17.48850148 podStartE2EDuration="17.48850148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:42.488446173 +0000 UTC m=+59.341150428" watchObservedRunningTime="2026-09-17 18:39:42.48850148 +0000 UTC m=+59.341205723"
Sep 17 18:39:43.831091 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:43.831058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:43.831419 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:43.831180    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:39:43.831419 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:43.831235    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert podName:3d8cb0ee-0042-48b9-91f4-3f29fee8230e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:59.831218702 +0000 UTC m=+76.683922926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-jcfxc" (UID: "3d8cb0ee-0042-48b9-91f4-3f29fee8230e") : secret "networking-console-plugin-cert" not found
Sep 17 18:39:44.470499 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:44.470434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6" event={"ID":"bfe11238-6e44-48c6-8129-762bf456a5c8","Type":"ContainerStarted","Data":"778a4a2f41246dd75df049d4ce9953b83c3ff0690f99facc0f525c57b1944f1f"}
Sep 17 18:39:44.470697 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:44.470557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6" event={"ID":"bfe11238-6e44-48c6-8129-762bf456a5c8","Type":"ContainerStarted","Data":"cac67181df2ab4e7b30489bd186fef65426f2fdb6b753b66a9a3f1c3804ed727"}
Sep 17 18:39:44.487601 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:44.487550    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-phwc6" podStartSLOduration=17.752351248 podStartE2EDuration="19.487533993s" totalImagesPullingTime="1.735182745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:44.487075513 +0000 UTC m=+61.339779758" watchObservedRunningTime="2026-09-17 18:39:44.487533993 +0000 UTC m=+61.340238237"
Sep 17 18:39:45.370105 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:45.370083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwfls"
Sep 17 18:39:46.347577 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:46.347526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:46.347577 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:46.347589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:46.347803 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:46.347696    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:46.347803 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:39:46.347751    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls podName:198c3a10-a9d4-40a5-b36a-d644cb3b548a nodeName:}" failed. No retries permitted until 2026-09-17 18:40:02.347736399 +0000 UTC m=+79.200440621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfkgq" (UID: "198c3a10-a9d4-40a5-b36a-d644cb3b548a") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:39:46.348086 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:46.348062    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/198c3a10-a9d4-40a5-b36a-d644cb3b548a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:39:50.481628 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.481597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:50.483792 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.483767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb-metrics-tls\") pod \"dns-default-khlll\" (UID: \"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb\") " pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:50.680508 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.680480    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zttzk\""
Sep 17 18:39:50.685717 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.685698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:50.884391 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.884364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:50.884531 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.884422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:50.886985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.886966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:50.887026 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.886987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:50.896970 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.896938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e4b8718-ff20-4f8b-bfee-853b19603fdc-metrics-certs\") pod \"network-metrics-daemon-9gtzs\" (UID: \"1e4b8718-ff20-4f8b-bfee-853b19603fdc\") " pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:50.897089 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.897072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/853ece52-4596-4c55-83af-da02fc283b14-original-pull-secret\") pod \"global-pull-secret-syncer-bvtmd\" (UID: \"853ece52-4596-4c55-83af-da02fc283b14\") " pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:50.956699 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:50.956675    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khlll"]
Sep 17 18:39:50.959238 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:50.959220    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98d1e82_ee9a_4887_b9d3_9cb3a054b3cb.slice/crio-a0f3e8046f4a5d6401033f32f0933b986829fbd9b6c6064b957f59ffba5c439a WatchSource:0}: Error finding container a0f3e8046f4a5d6401033f32f0933b986829fbd9b6c6064b957f59ffba5c439a: Status 404 returned error can't find the container with id a0f3e8046f4a5d6401033f32f0933b986829fbd9b6c6064b957f59ffba5c439a
Sep 17 18:39:51.003718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.003672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvtmd"
Sep 17 18:39:51.012296 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.012281    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kfmzw\""
Sep 17 18:39:51.020622 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.020606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9gtzs"
Sep 17 18:39:51.086385 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.086362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:51.089630 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.089078    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:39:51.100025 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.100005    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:39:51.109655 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.109638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxkl\" (UniqueName: \"kubernetes.io/projected/4e1e9522-f9b9-4675-a969-084168f1bb41-kube-api-access-6cxkl\") pod \"network-check-target-qfzqm\" (UID: \"4e1e9522-f9b9-4675-a969-084168f1bb41\") " pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:51.272660 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.272638    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bvtmd"]
Sep 17 18:39:51.275011 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:51.274989    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853ece52_4596_4c55_83af_da02fc283b14.slice/crio-af024f212039e2faa911994ad8cafe083207180a96082b94fb7003428c4f160b WatchSource:0}: Error finding container af024f212039e2faa911994ad8cafe083207180a96082b94fb7003428c4f160b: Status 404 returned error can't find the container with id af024f212039e2faa911994ad8cafe083207180a96082b94fb7003428c4f160b
Sep 17 18:39:51.294441 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.294418    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9gtzs"]
Sep 17 18:39:51.296708 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:51.296683    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4b8718_ff20_4f8b_bfee_853b19603fdc.slice/crio-e0da2f1dbff48fe44e3dd4a355398f9bbda13bc4455d7a011aff1a6224a23ac1 WatchSource:0}: Error finding container e0da2f1dbff48fe44e3dd4a355398f9bbda13bc4455d7a011aff1a6224a23ac1: Status 404 returned error can't find the container with id e0da2f1dbff48fe44e3dd4a355398f9bbda13bc4455d7a011aff1a6224a23ac1
Sep 17 18:39:51.316492 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.316474    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gk2t2\""
Sep 17 18:39:51.324941 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.324924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:51.494743 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.494700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khlll" event={"ID":"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb","Type":"ContainerStarted","Data":"a0f3e8046f4a5d6401033f32f0933b986829fbd9b6c6064b957f59ffba5c439a"}
Sep 17 18:39:51.496050 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.496018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9gtzs" event={"ID":"1e4b8718-ff20-4f8b-bfee-853b19603fdc","Type":"ContainerStarted","Data":"e0da2f1dbff48fe44e3dd4a355398f9bbda13bc4455d7a011aff1a6224a23ac1"}
Sep 17 18:39:51.497115 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.497090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bvtmd" event={"ID":"853ece52-4596-4c55-83af-da02fc283b14","Type":"ContainerStarted","Data":"af024f212039e2faa911994ad8cafe083207180a96082b94fb7003428c4f160b"}
Sep 17 18:39:51.595808 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:51.595788    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qfzqm"]
Sep 17 18:39:51.601783 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:51.601760    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1e9522_f9b9_4675_a969_084168f1bb41.slice/crio-cd3de6dea818ca1e8d990a9a9bea2720c6caef2989cb87bed76111a347d7b0f4 WatchSource:0}: Error finding container cd3de6dea818ca1e8d990a9a9bea2720c6caef2989cb87bed76111a347d7b0f4: Status 404 returned error can't find the container with id cd3de6dea818ca1e8d990a9a9bea2720c6caef2989cb87bed76111a347d7b0f4
Sep 17 18:39:52.502896 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:52.502856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qfzqm" event={"ID":"4e1e9522-f9b9-4675-a969-084168f1bb41","Type":"ContainerStarted","Data":"cd3de6dea818ca1e8d990a9a9bea2720c6caef2989cb87bed76111a347d7b0f4"}
Sep 17 18:39:53.511612 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.511038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9gtzs" event={"ID":"1e4b8718-ff20-4f8b-bfee-853b19603fdc","Type":"ContainerStarted","Data":"078b1bbdb873b6f5d6e3f0184091e461ea86384fdbabc45a5bf49eebe8bde231"}
Sep 17 18:39:53.511612 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.511118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9gtzs" event={"ID":"1e4b8718-ff20-4f8b-bfee-853b19603fdc","Type":"ContainerStarted","Data":"c287f8db84f8c32c9bbea97454895441c08b169c29584085882349720746c1cd"}
Sep 17 18:39:53.513533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.513503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khlll" event={"ID":"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb","Type":"ContainerStarted","Data":"734e04865e42e7d824b2ccb649d7bc89d1a9d04a7f1aa5b225ebefef097e2718"}
Sep 17 18:39:53.513640 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.513555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khlll" event={"ID":"d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb","Type":"ContainerStarted","Data":"466a7473ab1171bcb9d099b3bab9b1cbfabd74a7891b186d0a4896eab48f4565"}
Sep 17 18:39:53.513859 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.513815    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-khlll"
Sep 17 18:39:53.530720 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.530676    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9gtzs" podStartSLOduration=66.62254218 podStartE2EDuration="1m8.530664688s" totalImagesPullingTime="1.908122508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:53.52993412 +0000 UTC m=+70.382638366" watchObservedRunningTime="2026-09-17 18:39:53.530664688 +0000 UTC m=+70.383368932"
Sep 17 18:39:53.554168 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.553538    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-khlll" podStartSLOduration=33.303624254 podStartE2EDuration="35.553525666s" totalImagesPullingTime="2.249901412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:53.553281961 +0000 UTC m=+70.405986206" watchObservedRunningTime="2026-09-17 18:39:53.553525666 +0000 UTC m=+70.406229910"
Sep 17 18:39:53.997616 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:53.997578    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-ws6vr"]
Sep 17 18:39:54.000892 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.000871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:39:54.004138 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.004111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:39:54.005563 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.005107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:39:54.005563 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.005413    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2g4ql\""
Sep 17 18:39:54.015007 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.014976    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8476b68445-qgzvv"]
Sep 17 18:39:54.016050 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.016029    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-ws6vr"]
Sep 17 18:39:54.057661 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.057632    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c74b8f954-cgwhn"]
Sep 17 18:39:54.061121 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.061075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.084559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.084534    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c74b8f954-cgwhn"]
Sep 17 18:39:54.113206 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/107228bb-9cf8-425e-a692-f0fa85a8768e-registry-certificates\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113319 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/107228bb-9cf8-425e-a692-f0fa85a8768e-installation-pull-secrets\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113319 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwwnf\" (UniqueName: \"kubernetes.io/projected/36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5-kube-api-access-zwwnf\") pod \"downloads-6594f4c75-ws6vr\" (UID: \"36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5\") " pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:39:54.113530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/107228bb-9cf8-425e-a692-f0fa85a8768e-ca-trust-extracted\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-registry-tls\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/107228bb-9cf8-425e-a692-f0fa85a8768e-trusted-ca\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvz9\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-kube-api-access-cgvz9\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/107228bb-9cf8-425e-a692-f0fa85a8768e-image-registry-private-configuration\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.113724 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.113577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-bound-sa-token\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.214964 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.214929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvz9\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-kube-api-access-cgvz9\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.214964 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.214960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/107228bb-9cf8-425e-a692-f0fa85a8768e-image-registry-private-configuration\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.215202 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.214985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-bound-sa-token\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.215202 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.215046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/107228bb-9cf8-425e-a692-f0fa85a8768e-registry-certificates\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.215202 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.215086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/107228bb-9cf8-425e-a692-f0fa85a8768e-installation-pull-secrets\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.215202 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.215126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwwnf\" (UniqueName: \"kubernetes.io/projected/36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5-kube-api-access-zwwnf\") pod \"downloads-6594f4c75-ws6vr\" (UID: \"36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5\") " pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:39:54.215202 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.215194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/107228bb-9cf8-425e-a692-f0fa85a8768e-ca-trust-extracted\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.215481 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.215223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-registry-tls\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.215481 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.215248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/107228bb-9cf8-425e-a692-f0fa85a8768e-trusted-ca\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.216294 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.216109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/107228bb-9cf8-425e-a692-f0fa85a8768e-ca-trust-extracted\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.216294 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.216135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/107228bb-9cf8-425e-a692-f0fa85a8768e-registry-certificates\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.216294 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.216246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/107228bb-9cf8-425e-a692-f0fa85a8768e-trusted-ca\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.217912 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.217886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/107228bb-9cf8-425e-a692-f0fa85a8768e-image-registry-private-configuration\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.218062 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.218040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/107228bb-9cf8-425e-a692-f0fa85a8768e-installation-pull-secrets\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.218498 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.218478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-registry-tls\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.235706 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.235683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-bound-sa-token\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.237464 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.237433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvz9\" (UniqueName: \"kubernetes.io/projected/107228bb-9cf8-425e-a692-f0fa85a8768e-kube-api-access-cgvz9\") pod \"image-registry-7c74b8f954-cgwhn\" (UID: \"107228bb-9cf8-425e-a692-f0fa85a8768e\") " pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.237975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.237953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwwnf\" (UniqueName: \"kubernetes.io/projected/36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5-kube-api-access-zwwnf\") pod \"downloads-6594f4c75-ws6vr\" (UID: \"36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5\") " pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:39:54.312589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.312490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:39:54.371882 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.371642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:54.588944 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:54.588862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-ws6vr"]
Sep 17 18:39:56.651929 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:56.651890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d6cf8b_6e42_4dc2_bd08_dce0b6159ff5.slice/crio-5af8a90c89cce98f2fccbd81295df3a7f816c6c26245cd4db947cad5fdd79ab3 WatchSource:0}: Error finding container 5af8a90c89cce98f2fccbd81295df3a7f816c6c26245cd4db947cad5fdd79ab3: Status 404 returned error can't find the container with id 5af8a90c89cce98f2fccbd81295df3a7f816c6c26245cd4db947cad5fdd79ab3
Sep 17 18:39:56.972025 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:56.971973    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c74b8f954-cgwhn"]
Sep 17 18:39:56.974427 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:39:56.974400    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107228bb_9cf8_425e_a692_f0fa85a8768e.slice/crio-6a1f9405c136de1ce99046fc8570593455b6ed2a50f6a024d2bce97542d1df78 WatchSource:0}: Error finding container 6a1f9405c136de1ce99046fc8570593455b6ed2a50f6a024d2bce97542d1df78: Status 404 returned error can't find the container with id 6a1f9405c136de1ce99046fc8570593455b6ed2a50f6a024d2bce97542d1df78
Sep 17 18:39:57.529785 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.529749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qfzqm" event={"ID":"4e1e9522-f9b9-4675-a969-084168f1bb41","Type":"ContainerStarted","Data":"398952c101c1e0c188d4ddb363df7993f29e933f13b0baca981c53ad0e177afc"}
Sep 17 18:39:57.529975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.529823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:39:57.531429 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.531407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn" event={"ID":"107228bb-9cf8-425e-a692-f0fa85a8768e","Type":"ContainerStarted","Data":"3ddbc10432be1ad5f02b81e5c0b78993eb649ea3101132840841d1e2bf55e449"}
Sep 17 18:39:57.531542 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.531439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn" event={"ID":"107228bb-9cf8-425e-a692-f0fa85a8768e","Type":"ContainerStarted","Data":"6a1f9405c136de1ce99046fc8570593455b6ed2a50f6a024d2bce97542d1df78"}
Sep 17 18:39:57.531600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.531560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:39:57.532697 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.532676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-ws6vr" event={"ID":"36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5","Type":"ContainerStarted","Data":"5af8a90c89cce98f2fccbd81295df3a7f816c6c26245cd4db947cad5fdd79ab3"}
Sep 17 18:39:57.534344 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.534321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bvtmd" event={"ID":"853ece52-4596-4c55-83af-da02fc283b14","Type":"ContainerStarted","Data":"2f24732fc1823c21ec00f9181b1d8fcac2e616ce8c774d6847570b8396622559"}
Sep 17 18:39:57.553531 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.553488    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qfzqm" podStartSLOduration=67.449183097 podStartE2EDuration="1m12.553474791s" totalImagesPullingTime="5.104291694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:57.552887255 +0000 UTC m=+74.405591498" watchObservedRunningTime="2026-09-17 18:39:57.553474791 +0000 UTC m=+74.406179036"
Sep 17 18:39:57.572416 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.572376    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bvtmd" podStartSLOduration=67.133287282 podStartE2EDuration="1m12.572366159s" totalImagesPullingTime="5.439078877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:57.572121921 +0000 UTC m=+74.424826167" watchObservedRunningTime="2026-09-17 18:39:57.572366159 +0000 UTC m=+74.425070403"
Sep 17 18:39:57.594280 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:57.594243    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn" podStartSLOduration=3.5942327179999998 podStartE2EDuration="3.594232718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:57.593902045 +0000 UTC m=+74.446606289" watchObservedRunningTime="2026-09-17 18:39:57.594232718 +0000 UTC m=+74.446936962"
Sep 17 18:39:59.855341 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:59.855297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:39:59.858075 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:39:59.858042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d8cb0ee-0042-48b9-91f4-3f29fee8230e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jcfxc\" (UID: \"3d8cb0ee-0042-48b9-91f4-3f29fee8230e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:40:00.045550 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:00.045517    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nr5s7\""
Sep 17 18:40:00.053550 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:00.053526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"
Sep 17 18:40:00.328619 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:00.328586    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc"]
Sep 17 18:40:00.331569 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:00.331543    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8cb0ee_0042_48b9_91f4_3f29fee8230e.slice/crio-438883339ab87e8f95b1bbdaf8b679ed77a3fcc01af6b92ad7920b15174872c5 WatchSource:0}: Error finding container 438883339ab87e8f95b1bbdaf8b679ed77a3fcc01af6b92ad7920b15174872c5: Status 404 returned error can't find the container with id 438883339ab87e8f95b1bbdaf8b679ed77a3fcc01af6b92ad7920b15174872c5
Sep 17 18:40:00.547629 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:00.547547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc" event={"ID":"3d8cb0ee-0042-48b9-91f4-3f29fee8230e","Type":"ContainerStarted","Data":"438883339ab87e8f95b1bbdaf8b679ed77a3fcc01af6b92ad7920b15174872c5"}
Sep 17 18:40:02.375313 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.375277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:40:02.377777 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.377747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/198c3a10-a9d4-40a5-b36a-d644cb3b548a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfkgq\" (UID: \"198c3a10-a9d4-40a5-b36a-d644cb3b548a\") " pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:40:02.423139 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.423109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtkmh\""
Sep 17 18:40:02.430852 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.430816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nfkgq"
Sep 17 18:40:02.556779 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.556744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc" event={"ID":"3d8cb0ee-0042-48b9-91f4-3f29fee8230e","Type":"ContainerStarted","Data":"4f0ee7ae5d98d5192428ffcb045835614bf55a9e72464d42affdb8190e4ea651"}
Sep 17 18:40:02.580922 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.580875    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jcfxc" podStartSLOduration=34.308767838 podStartE2EDuration="35.580857898s" totalImagesPullingTime="1.27209006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:02.58020062 +0000 UTC m=+79.432904864" watchObservedRunningTime="2026-09-17 18:40:02.580857898 +0000 UTC m=+79.433562139"
Sep 17 18:40:02.712435 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:02.712389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nfkgq"]
Sep 17 18:40:02.715066 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:02.715040    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198c3a10_a9d4_40a5_b36a_d644cb3b548a.slice/crio-49704f069f570ebc4b894ae3d2052e5a83d99fe1d82a721229b7c96659315b6d WatchSource:0}: Error finding container 49704f069f570ebc4b894ae3d2052e5a83d99fe1d82a721229b7c96659315b6d: Status 404 returned error can't find the container with id 49704f069f570ebc4b894ae3d2052e5a83d99fe1d82a721229b7c96659315b6d
Sep 17 18:40:03.520297 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:03.520257    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-khlll"
Sep 17 18:40:03.562016 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:03.561978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfkgq" event={"ID":"198c3a10-a9d4-40a5-b36a-d644cb3b548a","Type":"ContainerStarted","Data":"8b27f4694dcc37cfbe41731e4bee5e0d72ea456254122adcd14ecd68c22c4903"}
Sep 17 18:40:03.562153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:03.562081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfkgq" event={"ID":"198c3a10-a9d4-40a5-b36a-d644cb3b548a","Type":"ContainerStarted","Data":"49704f069f570ebc4b894ae3d2052e5a83d99fe1d82a721229b7c96659315b6d"}
Sep 17 18:40:04.022622 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:04.022588    2566 patch_prober.go:28] interesting pod/image-registry-8476b68445-qgzvv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 18:40:04.022788 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:04.022653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" podUID="b33c6fa3-056b-490b-991a-afde594f8f62" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:04.567909 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:04.567875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfkgq" event={"ID":"198c3a10-a9d4-40a5-b36a-d644cb3b548a","Type":"ContainerStarted","Data":"58c703f4d43581a6fe6920c336ff7ce85367b631c6d20d0bfd9524eba32a96a8"}
Sep 17 18:40:05.572866 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:05.572772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfkgq" event={"ID":"198c3a10-a9d4-40a5-b36a-d644cb3b548a","Type":"ContainerStarted","Data":"e3d84447bf76fb5b4cef69ae592707bd5920c546f1387cddc20b66d40c3fb939"}
Sep 17 18:40:05.598844 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:05.598793    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nfkgq" podStartSLOduration=33.128891071 podStartE2EDuration="35.598777643s" totalImagesPullingTime="2.469886572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:05.596979704 +0000 UTC m=+82.449683949" watchObservedRunningTime="2026-09-17 18:40:05.598777643 +0000 UTC m=+82.451481910"
Sep 17 18:40:06.166219 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.166187    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"]
Sep 17 18:40:06.170703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.170680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.173368 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.173345    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:40:06.173467 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.173347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 18:40:06.174307 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.174140    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:40:06.174307 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.174179    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:40:06.174307 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.174201    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 18:40:06.174491 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.174308    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5v67p\""
Sep 17 18:40:06.182256 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.182236    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"]
Sep 17 18:40:06.306944 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.306906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.307143 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.306950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4adea632-e3b6-44f6-9bcc-ed0ce9862378-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.307143 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.306992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.307143 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.307080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thc2q\" (UniqueName: \"kubernetes.io/projected/4adea632-e3b6-44f6-9bcc-ed0ce9862378-kube-api-access-thc2q\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.408179 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.408157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thc2q\" (UniqueName: \"kubernetes.io/projected/4adea632-e3b6-44f6-9bcc-ed0ce9862378-kube-api-access-thc2q\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.408286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.408213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.408286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.408235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4adea632-e3b6-44f6-9bcc-ed0ce9862378-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.408286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.408267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.408448 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:40:06.408357    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 18:40:06.408448 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:40:06.408410    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-tls podName:4adea632-e3b6-44f6-9bcc-ed0ce9862378 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:06.908396305 +0000 UTC m=+83.761100540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-lp8m8" (UID: "4adea632-e3b6-44f6-9bcc-ed0ce9862378") : secret "prometheus-operator-tls" not found
Sep 17 18:40:06.408985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.408960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4adea632-e3b6-44f6-9bcc-ed0ce9862378-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.410516 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.410498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.418695 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.418650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thc2q\" (UniqueName: \"kubernetes.io/projected/4adea632-e3b6-44f6-9bcc-ed0ce9862378-kube-api-access-thc2q\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.911907 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.911877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:06.914446 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:06.914422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4adea632-e3b6-44f6-9bcc-ed0ce9862378-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-lp8m8\" (UID: \"4adea632-e3b6-44f6-9bcc-ed0ce9862378\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:07.080936 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:07.080908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"
Sep 17 18:40:07.363955 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:07.363818    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8"]
Sep 17 18:40:11.432012 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.431979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7975b49cc5-8wsfm"]
Sep 17 18:40:11.436822 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.436798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.439308 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.439287    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:40:11.439560 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.439541    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:40:11.440266 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.440239    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:40:11.440381 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.440290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hngcr\""
Sep 17 18:40:11.440808 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.440718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:40:11.440808 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.440720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:40:11.447532 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.447510    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:40:11.458418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.458390    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7975b49cc5-8wsfm"]
Sep 17 18:40:11.550840 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.550795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-config\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.551040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.550849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-trusted-ca-bundle\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.551040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.550876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-service-ca\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.551040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.550904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-oauth-serving-cert\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.551040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.550931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-oauth-config\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.551040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.550964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-serving-cert\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.551040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.551010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449nm\" (UniqueName: \"kubernetes.io/projected/f2ed9594-6d69-4658-b0de-99d26c8a8fce-kube-api-access-449nm\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.651862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.651811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-service-ca\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.651879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-oauth-serving-cert\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.651907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-oauth-config\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.651938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-serving-cert\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.651966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449nm\" (UniqueName: \"kubernetes.io/projected/f2ed9594-6d69-4658-b0de-99d26c8a8fce-kube-api-access-449nm\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652216 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.652036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-config\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652216 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.652065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-trusted-ca-bundle\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.652626 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.652598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-service-ca\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.653019 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.652996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-oauth-serving-cert\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.653094 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.653079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-trusted-ca-bundle\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.653199 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.653171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-config\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.655097 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.655078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-oauth-config\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.655097 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.655085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-serving-cert\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.666098 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.666077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-449nm\" (UniqueName: \"kubernetes.io/projected/f2ed9594-6d69-4658-b0de-99d26c8a8fce-kube-api-access-449nm\") pod \"console-7975b49cc5-8wsfm\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") " pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:11.748091 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:11.748032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:14.020743 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.020711    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:40:14.234051 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:14.234014    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adea632_e3b6_44f6_9bcc_ed0ce9862378.slice/crio-d6e9d5382b3e9f82cc013b959fe2ddc7378ad7ba856b245bb062fe1377a12761 WatchSource:0}: Error finding container d6e9d5382b3e9f82cc013b959fe2ddc7378ad7ba856b245bb062fe1377a12761: Status 404 returned error can't find the container with id d6e9d5382b3e9f82cc013b959fe2ddc7378ad7ba856b245bb062fe1377a12761
Sep 17 18:40:14.506230 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.506207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7975b49cc5-8wsfm"]
Sep 17 18:40:14.508862 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:14.508820    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ed9594_6d69_4658_b0de_99d26c8a8fce.slice/crio-c161e23fae212de11ef2900aced7e7706512b902bb484269c796df3c12460eb3 WatchSource:0}: Error finding container c161e23fae212de11ef2900aced7e7706512b902bb484269c796df3c12460eb3: Status 404 returned error can't find the container with id c161e23fae212de11ef2900aced7e7706512b902bb484269c796df3c12460eb3
Sep 17 18:40:14.610133 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.610082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8" event={"ID":"4adea632-e3b6-44f6-9bcc-ed0ce9862378","Type":"ContainerStarted","Data":"d6e9d5382b3e9f82cc013b959fe2ddc7378ad7ba856b245bb062fe1377a12761"}
Sep 17 18:40:14.611902 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.611876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-ws6vr" event={"ID":"36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5","Type":"ContainerStarted","Data":"ce5be9968af0ccfadb1f3342552599ed6f58de10b47c80db19638bb79e887e95"}
Sep 17 18:40:14.612304 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.612263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:40:14.613327 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.613279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975b49cc5-8wsfm" event={"ID":"f2ed9594-6d69-4658-b0de-99d26c8a8fce","Type":"ContainerStarted","Data":"c161e23fae212de11ef2900aced7e7706512b902bb484269c796df3c12460eb3"}
Sep 17 18:40:14.628069 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.628046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-ws6vr"
Sep 17 18:40:14.633407 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:14.633366    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-ws6vr" podStartSLOduration=3.990887249 podStartE2EDuration="21.633353087s" totalImagesPullingTime="17.642465838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:14.631098135 +0000 UTC m=+91.483802378" watchObservedRunningTime="2026-09-17 18:40:14.633353087 +0000 UTC m=+91.486057331"
Sep 17 18:40:18.544422 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:18.544387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c74b8f954-cgwhn"
Sep 17 18:40:19.039055 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:19.038970    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" podUID="b33c6fa3-056b-490b-991a-afde594f8f62" containerName="registry" containerID="cri-o://66a4f0be4f65f1a3e89425fdd116ebecc272c05f1f3f7e9773041467d17b2afd" gracePeriod=30
Sep 17 18:40:19.640793 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:19.640676    2566 generic.go:356] "Generic (PLEG): container finished" podID="b33c6fa3-056b-490b-991a-afde594f8f62" containerID="66a4f0be4f65f1a3e89425fdd116ebecc272c05f1f3f7e9773041467d17b2afd" exitCode=0
Sep 17 18:40:19.640793 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:19.640753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" event={"ID":"b33c6fa3-056b-490b-991a-afde594f8f62","Type":"ContainerDied","Data":"66a4f0be4f65f1a3e89425fdd116ebecc272c05f1f3f7e9773041467d17b2afd"}
Sep 17 18:40:20.596157 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.596136    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:40:20.646216 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.646187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8476b68445-qgzvv" event={"ID":"b33c6fa3-056b-490b-991a-afde594f8f62","Type":"ContainerDied","Data":"f7ec866ef2f1c4858b24a5de9de3e32e5d39e0d648d767bc7eb1326c66e39a7c"}
Sep 17 18:40:20.646581 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.646247    2566 scope.go:117] "RemoveContainer" containerID="66a4f0be4f65f1a3e89425fdd116ebecc272c05f1f3f7e9773041467d17b2afd"
Sep 17 18:40:20.646581 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.646407    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8476b68445-qgzvv"
Sep 17 18:40:20.738672 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-image-registry-private-configuration\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738797 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738715    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-registry-certificates\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738797 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738746    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-installation-pull-secrets\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738797 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jprr\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-kube-api-access-2jprr\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738994 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738994 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b33c6fa3-056b-490b-991a-afde594f8f62-ca-trust-extracted\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738994 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-trusted-ca\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.738994 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.738926    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-bound-sa-token\") pod \"b33c6fa3-056b-490b-991a-afde594f8f62\" (UID: \"b33c6fa3-056b-490b-991a-afde594f8f62\") "
Sep 17 18:40:20.739974 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.739722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:20.740088 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.740073    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:20.741810 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.741765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:20.742122 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.742094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-kube-api-access-2jprr" (OuterVolumeSpecName: "kube-api-access-2jprr") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "kube-api-access-2jprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:20.742578 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.742536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:20.743228 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.743191    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:20.743698 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.743666    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:20.750265 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.750234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33c6fa3-056b-490b-991a-afde594f8f62-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b33c6fa3-056b-490b-991a-afde594f8f62" (UID: "b33c6fa3-056b-490b-991a-afde594f8f62"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:20.840468 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840435    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-registry-certificates\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840468 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840470    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-installation-pull-secrets\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840658 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840484    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jprr\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-kube-api-access-2jprr\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840658 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840498    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-registry-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840658 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840512    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b33c6fa3-056b-490b-991a-afde594f8f62-ca-trust-extracted\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840658 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840524    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b33c6fa3-056b-490b-991a-afde594f8f62-trusted-ca\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840658 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840537    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b33c6fa3-056b-490b-991a-afde594f8f62-bound-sa-token\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.840658 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.840550    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b33c6fa3-056b-490b-991a-afde594f8f62-image-registry-private-configuration\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:40:20.973583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.973551    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8476b68445-qgzvv"]
Sep 17 18:40:20.978011 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:20.977988    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8476b68445-qgzvv"]
Sep 17 18:40:21.202245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.201909    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33c6fa3-056b-490b-991a-afde594f8f62" path="/var/lib/kubelet/pods/b33c6fa3-056b-490b-991a-afde594f8f62/volumes"
Sep 17 18:40:21.653750 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.653682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8" event={"ID":"4adea632-e3b6-44f6-9bcc-ed0ce9862378","Type":"ContainerStarted","Data":"daed1c3a05ef867fc769354c0d7b6e7bd06422ee8fb55e256e3b94bd246798b2"}
Sep 17 18:40:21.654220 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.653759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8" event={"ID":"4adea632-e3b6-44f6-9bcc-ed0ce9862378","Type":"ContainerStarted","Data":"af04e5cb42e1c7d1a32d6d3a202c0d9de4838485938810b241a7e2311b86aaf3"}
Sep 17 18:40:21.655917 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.655890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975b49cc5-8wsfm" event={"ID":"f2ed9594-6d69-4658-b0de-99d26c8a8fce","Type":"ContainerStarted","Data":"ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30"}
Sep 17 18:40:21.678969 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.678932    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-lp8m8" podStartSLOduration=9.362829102 podStartE2EDuration="15.678920608s" totalImagesPullingTime="6.316091506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:21.677895438 +0000 UTC m=+98.530599686" watchObservedRunningTime="2026-09-17 18:40:21.678920608 +0000 UTC m=+98.531624853"
Sep 17 18:40:21.748158 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.748123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:21.748273 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.748189    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:21.753453 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.753432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:21.787417 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:21.787375    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7975b49cc5-8wsfm" podStartSLOduration=4.241078126 podStartE2EDuration="10.787357429s" totalImagesPullingTime="6.546279303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:21.71499522 +0000 UTC m=+98.567699476" watchObservedRunningTime="2026-09-17 18:40:21.787357429 +0000 UTC m=+98.640061674"
Sep 17 18:40:22.663948 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:22.663917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:40:23.589179 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.589146    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"]
Sep 17 18:40:23.589399 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.589387    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33c6fa3-056b-490b-991a-afde594f8f62" containerName="registry"
Sep 17 18:40:23.589444 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.589401    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33c6fa3-056b-490b-991a-afde594f8f62" containerName="registry"
Sep 17 18:40:23.589496 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.589453    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33c6fa3-056b-490b-991a-afde594f8f62" containerName="registry"
Sep 17 18:40:23.621958 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.621933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"]
Sep 17 18:40:23.622110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.621982    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h7w59"]
Sep 17 18:40:23.622110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.622003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.625321 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.625298    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 18:40:23.625321 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.625312    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:40:23.625499 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.625313    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 18:40:23.625499 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.625299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-94s65\""
Sep 17 18:40:23.642068 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.642050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.644195 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.644177    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:40:23.644708 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.644692    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:40:23.644800 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.644787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:40:23.645435 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.645408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwb4x\""
Sep 17 18:40:23.663922 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.663902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-tls\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664022 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.663935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-sys\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664022 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.663959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.664022 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a7cdc2b-fe9c-483b-917b-bc76138e839e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.664357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a7cdc2b-fe9c-483b-917b-bc76138e839e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.664357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.664357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664357 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29cln\" (UniqueName: \"kubernetes.io/projected/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-api-access-29cln\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.664555 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dnr\" (UniqueName: \"kubernetes.io/projected/9d6c1b83-d256-498a-a829-91326947139c-kube-api-access-74dnr\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664555 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664555 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-textfile\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664708 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.664708 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-root\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664708 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d6c1b83-d256-498a-a829-91326947139c-metrics-client-ca\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.664708 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.664684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-wtmp\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.765737 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.765862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.765862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29cln\" (UniqueName: \"kubernetes.io/projected/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-api-access-29cln\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.765862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dnr\" (UniqueName: \"kubernetes.io/projected/9d6c1b83-d256-498a-a829-91326947139c-kube-api-access-74dnr\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.765862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.765862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-textfile\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-root\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d6c1b83-d256-498a-a829-91326947139c-metrics-client-ca\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.765982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-wtmp\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:40:23.765992    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-tls\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-sys\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-root\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:40:23.766057    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-tls podName:3a7cdc2b-fe9c-483b-917b-bc76138e839e nodeName:}" failed. No retries permitted until 2026-09-17 18:40:24.266038264 +0000 UTC m=+101.118742521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-z5jrk" (UID: "3a7cdc2b-fe9c-483b-917b-bc76138e839e") : secret "kube-state-metrics-tls" not found
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-sys\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.766141 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a7cdc2b-fe9c-483b-917b-bc76138e839e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.766687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a7cdc2b-fe9c-483b-917b-bc76138e839e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.766687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-wtmp\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a7cdc2b-fe9c-483b-917b-bc76138e839e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.766687 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d6c1b83-d256-498a-a829-91326947139c-metrics-client-ca\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.766956 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a7cdc2b-fe9c-483b-917b-bc76138e839e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.766956 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.766942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.767117 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.767090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-textfile\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.768642 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.768618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.769320 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.769301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.769320 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.769310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9d6c1b83-d256-498a-a829-91326947139c-node-exporter-tls\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.781948 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.781928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dnr\" (UniqueName: \"kubernetes.io/projected/9d6c1b83-d256-498a-a829-91326947139c-kube-api-access-74dnr\") pod \"node-exporter-h7w59\" (UID: \"9d6c1b83-d256-498a-a829-91326947139c\") " pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.783585 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.783564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cln\" (UniqueName: \"kubernetes.io/projected/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-api-access-29cln\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:23.950561 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:23.950505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7w59"
Sep 17 18:40:23.998722 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:23.998700    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6c1b83_d256_498a_a829_91326947139c.slice/crio-107a7622413ab3f86d231dea23acb75e0c88620a4fcce2e69c4780345b38a5d3 WatchSource:0}: Error finding container 107a7622413ab3f86d231dea23acb75e0c88620a4fcce2e69c4780345b38a5d3: Status 404 returned error can't find the container with id 107a7622413ab3f86d231dea23acb75e0c88620a4fcce2e69c4780345b38a5d3
Sep 17 18:40:24.270174 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:24.270113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:24.272406 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:24.272387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a7cdc2b-fe9c-483b-917b-bc76138e839e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z5jrk\" (UID: \"3a7cdc2b-fe9c-483b-917b-bc76138e839e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:24.531972 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:24.531904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"
Sep 17 18:40:24.669134 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:24.669077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7w59" event={"ID":"9d6c1b83-d256-498a-a829-91326947139c","Type":"ContainerStarted","Data":"107a7622413ab3f86d231dea23acb75e0c88620a4fcce2e69c4780345b38a5d3"}
Sep 17 18:40:24.828130 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:24.828103    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-z5jrk"]
Sep 17 18:40:24.830555 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:24.830523    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7cdc2b_fe9c_483b_917b_bc76138e839e.slice/crio-661052f2e50052589c97b999b21309aefcdaabc2ddcd417677a694e028e53409 WatchSource:0}: Error finding container 661052f2e50052589c97b999b21309aefcdaabc2ddcd417677a694e028e53409: Status 404 returned error can't find the container with id 661052f2e50052589c97b999b21309aefcdaabc2ddcd417677a694e028e53409
Sep 17 18:40:25.676420 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:25.676378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7w59" event={"ID":"9d6c1b83-d256-498a-a829-91326947139c","Type":"ContainerStarted","Data":"b2bb9085fe24a36efcab82a68940577c5e20b941c4182e877d3d94de911cd042"}
Sep 17 18:40:25.678230 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:25.678191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk" event={"ID":"3a7cdc2b-fe9c-483b-917b-bc76138e839e","Type":"ContainerStarted","Data":"661052f2e50052589c97b999b21309aefcdaabc2ddcd417677a694e028e53409"}
Sep 17 18:40:26.683857 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:26.683813    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d6c1b83-d256-498a-a829-91326947139c" containerID="b2bb9085fe24a36efcab82a68940577c5e20b941c4182e877d3d94de911cd042" exitCode=0
Sep 17 18:40:26.684232 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:26.683877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7w59" event={"ID":"9d6c1b83-d256-498a-a829-91326947139c","Type":"ContainerDied","Data":"b2bb9085fe24a36efcab82a68940577c5e20b941c4182e877d3d94de911cd042"}
Sep 17 18:40:27.689758 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.689721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7w59" event={"ID":"9d6c1b83-d256-498a-a829-91326947139c","Type":"ContainerStarted","Data":"153c8935a095507f76fb78634ea97c11e587e37b36e4b0a7225d1c170faef62d"}
Sep 17 18:40:27.690244 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.689776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7w59" event={"ID":"9d6c1b83-d256-498a-a829-91326947139c","Type":"ContainerStarted","Data":"caea7192ff969c1f96cce1f92cd7df081e0e96568195011714f0ee57df44443f"}
Sep 17 18:40:27.692112 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.692083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk" event={"ID":"3a7cdc2b-fe9c-483b-917b-bc76138e839e","Type":"ContainerStarted","Data":"abb0313a5e8629dfbb113142cb16f342fb92ed78017704173b129d5516826f25"}
Sep 17 18:40:27.692243 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.692132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk" event={"ID":"3a7cdc2b-fe9c-483b-917b-bc76138e839e","Type":"ContainerStarted","Data":"42c801cc08fc0ba152a6cc5325ba0e17559f61b3caa3335593473ea5766c8789"}
Sep 17 18:40:27.692243 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.692150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk" event={"ID":"3a7cdc2b-fe9c-483b-917b-bc76138e839e","Type":"ContainerStarted","Data":"ed4bba226c6b03902f9fc224e2ca26450d3497fc598f6b342e434d60d6578c72"}
Sep 17 18:40:27.725086 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.725032    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h7w59" podStartSLOduration=3.123284 podStartE2EDuration="4.725013917s" totalImagesPullingTime="1.519583s" totalInitContainerRuntime="82.146917ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:27.724033091 +0000 UTC m=+104.576737335" watchObservedRunningTime="2026-09-17 18:40:27.725013917 +0000 UTC m=+104.577718164"
Sep 17 18:40:27.765412 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.765364    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-z5jrk" podStartSLOduration=2.755073234 podStartE2EDuration="4.765343874s" totalImagesPullingTime="2.01027064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:27.764485957 +0000 UTC m=+104.617190202" watchObservedRunningTime="2026-09-17 18:40:27.765343874 +0000 UTC m=+104.618048118"
Sep 17 18:40:27.868440 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.868411    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"]
Sep 17 18:40:27.888029 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.888005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"]
Sep 17 18:40:27.888143 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.888104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.890795 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.890771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1i96qm5p14j4e\""
Sep 17 18:40:27.890979 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.890810    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 18:40:27.890979 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.890782    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 18:40:27.890979 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.890911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 18:40:27.890979 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.890975    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7wdf5\""
Sep 17 18:40:27.891168 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.891129    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 18:40:27.900942 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.900924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f99c367-a9cb-4657-8495-7b144b21737b-audit-log\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.901029 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.900960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-secret-metrics-server-client-certs\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.901029 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.900980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f99c367-a9cb-4657-8495-7b144b21737b-metrics-server-audit-profiles\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.901099 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.901025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62j2\" (UniqueName: \"kubernetes.io/projected/9f99c367-a9cb-4657-8495-7b144b21737b-kube-api-access-t62j2\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.901099 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.901061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-secret-metrics-server-tls\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.901163 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.901099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-client-ca-bundle\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:27.901163 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:27.901118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f99c367-a9cb-4657-8495-7b144b21737b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002065 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-secret-metrics-server-tls\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002065 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-client-ca-bundle\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002215 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f99c367-a9cb-4657-8495-7b144b21737b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002215 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f99c367-a9cb-4657-8495-7b144b21737b-audit-log\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002319 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-secret-metrics-server-client-certs\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002319 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f99c367-a9cb-4657-8495-7b144b21737b-metrics-server-audit-profiles\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002426 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62j2\" (UniqueName: \"kubernetes.io/projected/9f99c367-a9cb-4657-8495-7b144b21737b-kube-api-access-t62j2\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002611 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f99c367-a9cb-4657-8495-7b144b21737b-audit-log\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.002718 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.002700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f99c367-a9cb-4657-8495-7b144b21737b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.003413 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.003387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f99c367-a9cb-4657-8495-7b144b21737b-metrics-server-audit-profiles\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.004422 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.004395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-secret-metrics-server-tls\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.004591 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.004570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-secret-metrics-server-client-certs\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.004671 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.004652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f99c367-a9cb-4657-8495-7b144b21737b-client-ca-bundle\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.011824 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.011802    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62j2\" (UniqueName: \"kubernetes.io/projected/9f99c367-a9cb-4657-8495-7b144b21737b-kube-api-access-t62j2\") pod \"metrics-server-5bb5cf5c69-nx7wl\" (UID: \"9f99c367-a9cb-4657-8495-7b144b21737b\") " pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.197488 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.197461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:28.354699 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.354666    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"]
Sep 17 18:40:28.376627 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.376607    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"]
Sep 17 18:40:28.376759 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.376697    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.383114 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.383091    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 18:40:28.383245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.383216    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vp54k\""
Sep 17 18:40:28.405169 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.405148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fdfbf636-a10e-4384-b47c-efdadeafdb88-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-6wdp7\" (UID: \"fdfbf636-a10e-4384-b47c-efdadeafdb88\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.405274 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.405188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rhs\" (UniqueName: \"kubernetes.io/projected/fdfbf636-a10e-4384-b47c-efdadeafdb88-kube-api-access-42rhs\") pod \"monitoring-plugin-5984459ff8-6wdp7\" (UID: \"fdfbf636-a10e-4384-b47c-efdadeafdb88\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.500617 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.500587    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"]
Sep 17 18:40:28.503972 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:28.503946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f99c367_a9cb_4657_8495_7b144b21737b.slice/crio-534a8c30c5a47a1c4afa3ad2f1ace93c982d445068c54287e6e3d5b2770cb083 WatchSource:0}: Error finding container 534a8c30c5a47a1c4afa3ad2f1ace93c982d445068c54287e6e3d5b2770cb083: Status 404 returned error can't find the container with id 534a8c30c5a47a1c4afa3ad2f1ace93c982d445068c54287e6e3d5b2770cb083
Sep 17 18:40:28.505740 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.505647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fdfbf636-a10e-4384-b47c-efdadeafdb88-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-6wdp7\" (UID: \"fdfbf636-a10e-4384-b47c-efdadeafdb88\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.505740 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.505697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rhs\" (UniqueName: \"kubernetes.io/projected/fdfbf636-a10e-4384-b47c-efdadeafdb88-kube-api-access-42rhs\") pod \"monitoring-plugin-5984459ff8-6wdp7\" (UID: \"fdfbf636-a10e-4384-b47c-efdadeafdb88\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.508253 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.508233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fdfbf636-a10e-4384-b47c-efdadeafdb88-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-6wdp7\" (UID: \"fdfbf636-a10e-4384-b47c-efdadeafdb88\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.521846 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.521775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rhs\" (UniqueName: \"kubernetes.io/projected/fdfbf636-a10e-4384-b47c-efdadeafdb88-kube-api-access-42rhs\") pod \"monitoring-plugin-5984459ff8-6wdp7\" (UID: \"fdfbf636-a10e-4384-b47c-efdadeafdb88\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.541794 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.541769    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qfzqm"
Sep 17 18:40:28.685808 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.685780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:28.697330 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.697299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl" event={"ID":"9f99c367-a9cb-4657-8495-7b144b21737b","Type":"ContainerStarted","Data":"534a8c30c5a47a1c4afa3ad2f1ace93c982d445068c54287e6e3d5b2770cb083"}
Sep 17 18:40:28.974602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:28.974573    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"]
Sep 17 18:40:28.977809 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:28.977784    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfbf636_a10e_4384_b47c_efdadeafdb88.slice/crio-b893afa30ccb36ba11a1be919f64e8be79de9c4f8bd0e5ddeea4f0631d8a5442 WatchSource:0}: Error finding container b893afa30ccb36ba11a1be919f64e8be79de9c4f8bd0e5ddeea4f0631d8a5442: Status 404 returned error can't find the container with id b893afa30ccb36ba11a1be919f64e8be79de9c4f8bd0e5ddeea4f0631d8a5442
Sep 17 18:40:29.704276 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.704245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7" event={"ID":"fdfbf636-a10e-4384-b47c-efdadeafdb88","Type":"ContainerStarted","Data":"b893afa30ccb36ba11a1be919f64e8be79de9c4f8bd0e5ddeea4f0631d8a5442"}
Sep 17 18:40:29.847204 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.847157    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:40:29.877361 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.876039    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:40:29.877361 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.876234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.880725 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.880536    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:40:29.880862 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.880772    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.884083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.884273    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.884471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.884799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-k69mb2f61s1b\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.884848    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.885049    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.885321    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.885415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.885496    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.885601    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:40:29.886400 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.885896    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:40:29.887036 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.886956    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k79wc\""
Sep 17 18:40:29.888717 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.888680    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:40:29.904367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.904335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919306    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.919786 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.920776 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.920776 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwfs\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-kube-api-access-shwfs\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.920776 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:29.920776 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:29.919738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwfs\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-kube-api-access-shwfs\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.020739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.021286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.021286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.021286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.021286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.021286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.021286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.020947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.022542 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.022503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.023170 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.023142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.024068 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.023958    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.026674 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.026225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.026674 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.026418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.027997 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.027540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.029218 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.028393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.029906 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.029634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.032589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.030029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.032589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.031055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.032589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.031139    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.032589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.031565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.032589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.032143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.033513 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.033220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.033513 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.033262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.033513 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.033301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.034475 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.033745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.035081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.035030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.039603 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.035464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.039603 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.036148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwfs\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-kube-api-access-shwfs\") pod \"prometheus-k8s-0\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.194820 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.194792    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:30.518007 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:30.517738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:40:30.924272 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:40:30.924240    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6507bd_e6f0_41de_851f_8e2f7b87acbd.slice/crio-20221ae19d89c939b15c180b0c7c7a4590e15115c5cd90559663ff71333da4aa WatchSource:0}: Error finding container 20221ae19d89c939b15c180b0c7c7a4590e15115c5cd90559663ff71333da4aa: Status 404 returned error can't find the container with id 20221ae19d89c939b15c180b0c7c7a4590e15115c5cd90559663ff71333da4aa
Sep 17 18:40:31.718143 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.718093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl" event={"ID":"9f99c367-a9cb-4657-8495-7b144b21737b","Type":"ContainerStarted","Data":"863cce163398fef66ade58ba1dd71aa58412890b0adde6a98cdf1547251ad3d0"}
Sep 17 18:40:31.719541 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.719510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"20221ae19d89c939b15c180b0c7c7a4590e15115c5cd90559663ff71333da4aa"}
Sep 17 18:40:31.721263 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.721240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7" event={"ID":"fdfbf636-a10e-4384-b47c-efdadeafdb88","Type":"ContainerStarted","Data":"d532a8c8edabb900d7048222d9b7abdf7331c3c77743de919a9e24fa98203454"}
Sep 17 18:40:31.721475 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.721452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:31.727444 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.727423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7"
Sep 17 18:40:31.737041 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.737003    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl" podStartSLOduration=2.099866633 podStartE2EDuration="4.736988872s" totalImagesPullingTime="2.637122239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:31.735608074 +0000 UTC m=+108.588312320" watchObservedRunningTime="2026-09-17 18:40:31.736988872 +0000 UTC m=+108.589693115"
Sep 17 18:40:31.754967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:31.754882    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6wdp7" podStartSLOduration=1.588738287 podStartE2EDuration="3.754869228s" totalImagesPullingTime="2.166130941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:31.752734835 +0000 UTC m=+108.605439082" watchObservedRunningTime="2026-09-17 18:40:31.754869228 +0000 UTC m=+108.607573472"
Sep 17 18:40:32.726890 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:32.726863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"}
Sep 17 18:40:33.733039 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:33.733006    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf" exitCode=0
Sep 17 18:40:33.733418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:33.733100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"}
Sep 17 18:40:37.753121 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:37.753086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"}
Sep 17 18:40:37.753475 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:37.753154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"}
Sep 17 18:40:38.798959 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:38.798923    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7975b49cc5-8wsfm"]
Sep 17 18:40:40.769992 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:40.769966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"}
Sep 17 18:40:40.770278 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:40.770064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"}
Sep 17 18:40:40.770278 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:40.770094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"}
Sep 17 18:40:40.770278 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:40.770116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerStarted","Data":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"}
Sep 17 18:40:41.808512 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:41.808451    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.071098198 podStartE2EDuration="12.808428419s" totalImagesPullingTime="7.716430599s" totalInitContainerRuntime="20.899622ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:41.806778011 +0000 UTC m=+118.659482255" watchObservedRunningTime="2026-09-17 18:40:41.808428419 +0000 UTC m=+118.661132664"
Sep 17 18:40:45.198406 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:45.198376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:40:48.198137 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:48.198104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:48.198520 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:48.198150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:40:59.853984 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:59.853956    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab21b025-a42f-4bfb-bbe2-16ab6d0d5109" containerID="c77feb24540513cff4c91a996b14a1c4848d88a7643e536c4c87d3f462e73886" exitCode=0
Sep 17 18:40:59.854280 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:59.854010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99" event={"ID":"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109","Type":"ContainerDied","Data":"c77feb24540513cff4c91a996b14a1c4848d88a7643e536c4c87d3f462e73886"}
Sep 17 18:40:59.854280 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:40:59.854254    2566 scope.go:117] "RemoveContainer" containerID="c77feb24540513cff4c91a996b14a1c4848d88a7643e536c4c87d3f462e73886"
Sep 17 18:41:00.860129 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:00.860092    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hkp99" event={"ID":"ab21b025-a42f-4bfb-bbe2-16ab6d0d5109","Type":"ContainerStarted","Data":"f1ad3dddefb287042c671df8077dbb41b19b2896e7f74760c9940ed53c8a9bee"}
Sep 17 18:41:03.825692 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:03.825625    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7975b49cc5-8wsfm" podUID="f2ed9594-6d69-4658-b0de-99d26c8a8fce" containerName="console" containerID="cri-o://ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30" gracePeriod=15
Sep 17 18:41:04.102225 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.102162    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:41:04.183820 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.183791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-service-ca\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.183939 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.183851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-serving-cert\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.184066 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-trusted-ca-bundle\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.184139 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184124    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-oauth-serving-cert\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.184196 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-449nm\" (UniqueName: \"kubernetes.io/projected/f2ed9594-6d69-4658-b0de-99d26c8a8fce-kube-api-access-449nm\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.184251 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-oauth-config\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.184306 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-config\") pod \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\" (UID: \"f2ed9594-6d69-4658-b0de-99d26c8a8fce\") "
Sep 17 18:41:04.184566 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-service-ca" (OuterVolumeSpecName: "service-ca") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:04.184566 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184386    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:04.184566 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184496    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:04.184788 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.184765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-config" (OuterVolumeSpecName: "console-config") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:04.186160 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.186135    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:04.186589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.186563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ed9594-6d69-4658-b0de-99d26c8a8fce-kube-api-access-449nm" (OuterVolumeSpecName: "kube-api-access-449nm") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "kube-api-access-449nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:04.186748 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.186731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f2ed9594-6d69-4658-b0de-99d26c8a8fce" (UID: "f2ed9594-6d69-4658-b0de-99d26c8a8fce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:04.285764 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285736    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-oauth-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.285764 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285762    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.285938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285774    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-service-ca\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.285938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285786    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ed9594-6d69-4658-b0de-99d26c8a8fce-console-serving-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.285938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285799    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-trusted-ca-bundle\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.285938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285811    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed9594-6d69-4658-b0de-99d26c8a8fce-oauth-serving-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.285938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.285823    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449nm\" (UniqueName: \"kubernetes.io/projected/f2ed9594-6d69-4658-b0de-99d26c8a8fce-kube-api-access-449nm\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:04.877469 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.877434    2566 generic.go:356] "Generic (PLEG): container finished" podID="d81bfd12-ec20-4803-b867-dc1401885094" containerID="f544a42418fe633e594110c06a5a4391eddcfaf410439e839f9327048b8cc839" exitCode=0
Sep 17 18:41:04.877935 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.877506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf" event={"ID":"d81bfd12-ec20-4803-b867-dc1401885094","Type":"ContainerDied","Data":"f544a42418fe633e594110c06a5a4391eddcfaf410439e839f9327048b8cc839"}
Sep 17 18:41:04.877935 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.877868    2566 scope.go:117] "RemoveContainer" containerID="f544a42418fe633e594110c06a5a4391eddcfaf410439e839f9327048b8cc839"
Sep 17 18:41:04.879256 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.879232    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2ed9594-6d69-4658-b0de-99d26c8a8fce" containerID="ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30" exitCode=0
Sep 17 18:41:04.879360 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.879279    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975b49cc5-8wsfm"
Sep 17 18:41:04.879360 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.879313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975b49cc5-8wsfm" event={"ID":"f2ed9594-6d69-4658-b0de-99d26c8a8fce","Type":"ContainerDied","Data":"ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30"}
Sep 17 18:41:04.879430 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.879363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975b49cc5-8wsfm" event={"ID":"f2ed9594-6d69-4658-b0de-99d26c8a8fce","Type":"ContainerDied","Data":"c161e23fae212de11ef2900aced7e7706512b902bb484269c796df3c12460eb3"}
Sep 17 18:41:04.879430 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.879386    2566 scope.go:117] "RemoveContainer" containerID="ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30"
Sep 17 18:41:04.891723 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.891695    2566 scope.go:117] "RemoveContainer" containerID="ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30"
Sep 17 18:41:04.892033 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:04.891991    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30\": container with ID starting with ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30 not found: ID does not exist" containerID="ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30"
Sep 17 18:41:04.892107 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.892040    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30"} err="failed to get container status \"ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30\": rpc error: code = NotFound desc = could not find container \"ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30\": container with ID starting with ba190572095211ad33a6c4f265350e1d889a36e0df8f743f1a0659b02ea18f30 not found: ID does not exist"
Sep 17 18:41:04.913060 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.913033    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7975b49cc5-8wsfm"]
Sep 17 18:41:04.919047 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:04.919014    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7975b49cc5-8wsfm"]
Sep 17 18:41:05.199189 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:05.199108    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ed9594-6d69-4658-b0de-99d26c8a8fce" path="/var/lib/kubelet/pods/f2ed9594-6d69-4658-b0de-99d26c8a8fce/volumes"
Sep 17 18:41:05.885242 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:05.885209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jfxvf" event={"ID":"d81bfd12-ec20-4803-b867-dc1401885094","Type":"ContainerStarted","Data":"84b54e3bc8885769c8a132463d4cbd57a2d8cb7b31b663f11000afb180335594"}
Sep 17 18:41:08.203655 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:08.203623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:41:08.207518 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:08.207495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bb5cf5c69-nx7wl"
Sep 17 18:41:30.195793 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:30.195710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:30.213842 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:30.213814    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:31.008124 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:31.008097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:48.201850 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.201802    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:48.202312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.202165    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="prometheus" containerID="cri-o://1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc" gracePeriod=600
Sep 17 18:41:48.202312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.202179    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy" containerID="cri-o://c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26" gracePeriod=600
Sep 17 18:41:48.202312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.202214    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="config-reloader" containerID="cri-o://186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e" gracePeriod=600
Sep 17 18:41:48.202312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.202267    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-web" containerID="cri-o://caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd" gracePeriod=600
Sep 17 18:41:48.202312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.202215    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="thanos-sidecar" containerID="cri-o://c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2" gracePeriod=600
Sep 17 18:41:48.202570 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.202342    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1" gracePeriod=600
Sep 17 18:41:48.462404 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.462346    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518058    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-kube-rbac-proxy\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518107    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-web-config\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-tls\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518193    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-kubelet-serving-ca-bundle\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518253    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-2\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518283    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shwfs\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-kube-api-access-shwfs\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-trusted-ca-bundle\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-metrics-client-ca\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-0\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518421    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-grpc-tls\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-metrics-client-certs\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-thanos-prometheus-http-client-file\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518537    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-tls-assets\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-1\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-serving-certs-ca-bundle\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.523367 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518624    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-db\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518637    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.518876    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.519444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.519716    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.519987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.521180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config" (OuterVolumeSpecName: "config") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.521659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.521714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.521788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.522220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.522362    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.522556    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.522747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:48.524599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.523318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.525451 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.523603    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.525877 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.525854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.525877 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.525870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-kube-api-access-shwfs" (OuterVolumeSpecName: "kube-api-access-shwfs") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "kube-api-access-shwfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:48.527133 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.527111    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:48.537799 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.537778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-web-config" (OuterVolumeSpecName: "web-config") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.619320 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.619426 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619329    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config-out\") pod \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\" (UID: \"ac6507bd-e6f0-41de-851f-8e2f7b87acbd\") "
Sep 17 18:41:48.619473 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619453    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619473 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619463    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-tls-assets\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619473 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619472    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619480    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619489    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-db\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619498    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-kube-rbac-proxy\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619506    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619515    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-web-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619522    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619530    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619538    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619545    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwfs\" (UniqueName: \"kubernetes.io/projected/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-kube-api-access-shwfs\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619554    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619562    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-configmap-metrics-client-ca\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619571    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619579    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-grpc-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.619583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.619587    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-metrics-client-certs\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.621082 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.621054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:48.621170 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.621153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config-out" (OuterVolumeSpecName: "config-out") pod "ac6507bd-e6f0-41de-851f-8e2f7b87acbd" (UID: "ac6507bd-e6f0-41de-851f-8e2f7b87acbd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:48.720591 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.720525    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:48.720591 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:48.720559    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6507bd-e6f0-41de-851f-8e2f7b87acbd-config-out\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:41:49.069424 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069397    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1" exitCode=0
Sep 17 18:41:49.069424 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069420    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26" exitCode=0
Sep 17 18:41:49.069424 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069428    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd" exitCode=0
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069434    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2" exitCode=0
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069442    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e" exitCode=0
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069449    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc" exitCode=0
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"}
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"}
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069493    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.069600 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"}
Sep 17 18:41:49.070001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"}
Sep 17 18:41:49.070001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069654    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.070001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"}
Sep 17 18:41:49.070001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"}
Sep 17 18:41:49.070001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.069768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac6507bd-e6f0-41de-851f-8e2f7b87acbd","Type":"ContainerDied","Data":"20221ae19d89c939b15c180b0c7c7a4590e15115c5cd90559663ff71333da4aa"}
Sep 17 18:41:49.082741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.079982    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.090209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.090192    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.097095 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.097075    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.100530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.100502    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:49.105405 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.105380    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:49.106037 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.106015    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.112807 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.112793    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.119845 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.119806    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.128311 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.128293    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.128588 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.128571    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.128640 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.128597    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"} err="failed to get container status \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist"
Sep 17 18:41:49.128640 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.128615    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.128906 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.128887    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.128967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.128912    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"} err="failed to get container status \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist"
Sep 17 18:41:49.128967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.128932    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.129198 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.129183    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.129250 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129198    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"} err="failed to get container status \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist"
Sep 17 18:41:49.129250 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129210    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.129435 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.129416    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.129474 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129438    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"} err="failed to get container status \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist"
Sep 17 18:41:49.129474 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129452    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.129690 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.129675    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.129803 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129693    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"} err="failed to get container status \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist"
Sep 17 18:41:49.129803 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129709    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.129962 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.129945    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.130008 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129966    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"} err="failed to get container status \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist"
Sep 17 18:41:49.130008 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.129979    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.130200 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:41:49.130185    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.130242 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.130202    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"} err="failed to get container status \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist"
Sep 17 18:41:49.130242 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.130214    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.130524 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.130462    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"} err="failed to get container status \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist"
Sep 17 18:41:49.130524 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.130488    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.131015 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.130988    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"} err="failed to get container status \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist"
Sep 17 18:41:49.131015 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.131017    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.131388 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.131365    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"} err="failed to get container status \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist"
Sep 17 18:41:49.131482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.131391    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.131690 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.131666    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"} err="failed to get container status \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist"
Sep 17 18:41:49.131758 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.131691    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.132006 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.131980    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"} err="failed to get container status \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist"
Sep 17 18:41:49.132114 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132011    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.132276 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132254    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"} err="failed to get container status \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist"
Sep 17 18:41:49.132330 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132276    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.132365 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:49.132539 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132520    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"} err="failed to get container status \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist"
Sep 17 18:41:49.132584 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132540    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.132646 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132632    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ed9594-6d69-4658-b0de-99d26c8a8fce" containerName="console"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132648    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ed9594-6d69-4658-b0de-99d26c8a8fce" containerName="console"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132659    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="config-reloader"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132665    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="config-reloader"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132675    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-thanos"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132684    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-thanos"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132694    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-web"
Sep 17 18:41:49.132703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132700    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-web"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132712    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132720    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132730    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="init-config-reloader"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132736    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="init-config-reloader"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132746    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="prometheus"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132752    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="prometheus"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132758    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="thanos-sidecar"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132767    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="thanos-sidecar"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132770    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"} err="failed to get container status \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132786    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132818    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="thanos-sidecar"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132841    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="config-reloader"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132851    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132861    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ed9594-6d69-4658-b0de-99d26c8a8fce" containerName="console"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132869    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-web"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132878    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="kube-rbac-proxy-thanos"
Sep 17 18:41:49.132989 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.132886    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" containerName="prometheus"
Sep 17 18:41:49.133602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133065    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"} err="failed to get container status \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist"
Sep 17 18:41:49.133602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133081    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.133602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133327    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"} err="failed to get container status \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist"
Sep 17 18:41:49.133602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133341    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.133602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133567    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"} err="failed to get container status \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist"
Sep 17 18:41:49.133602 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133589    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.133879 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133815    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"} err="failed to get container status \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist"
Sep 17 18:41:49.133931 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.133882    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.134148 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134122    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"} err="failed to get container status \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist"
Sep 17 18:41:49.134211 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134150    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.134395 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134377    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"} err="failed to get container status \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist"
Sep 17 18:41:49.134441 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134402    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.134649 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134633    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"} err="failed to get container status \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist"
Sep 17 18:41:49.134692 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134649    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.134939 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134920    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"} err="failed to get container status \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist"
Sep 17 18:41:49.134994 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.134940    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.135154 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135138    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"} err="failed to get container status \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist"
Sep 17 18:41:49.135192 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135153    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.135393 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135372    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"} err="failed to get container status \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist"
Sep 17 18:41:49.135452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135394    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.135634 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135616    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"} err="failed to get container status \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist"
Sep 17 18:41:49.135691 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135634    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.135860 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135816    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"} err="failed to get container status \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist"
Sep 17 18:41:49.135860 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.135852    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.136092 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136072    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"} err="failed to get container status \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist"
Sep 17 18:41:49.136145 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136092    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.136312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136295    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"} err="failed to get container status \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist"
Sep 17 18:41:49.136312 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136311    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.136541 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136523    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"} err="failed to get container status \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist"
Sep 17 18:41:49.136582 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136542    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.136754 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136734    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"} err="failed to get container status \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist"
Sep 17 18:41:49.136754 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.136750    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.137127 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137096    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"} err="failed to get container status \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist"
Sep 17 18:41:49.137127 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137122    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.137452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137429    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"} err="failed to get container status \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist"
Sep 17 18:41:49.137452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137451    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.137704 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137681    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"} err="failed to get container status \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist"
Sep 17 18:41:49.137704 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137702    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.138002 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.137980    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"} err="failed to get container status \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist"
Sep 17 18:41:49.138059 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138004    2566 scope.go:117] "RemoveContainer" containerID="241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"
Sep 17 18:41:49.138292 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138270    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1"} err="failed to get container status \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": rpc error: code = NotFound desc = could not find container \"241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1\": container with ID starting with 241e66323183c8217de424b158a088ec50ccb2f0a4c63bedca8d2f55a1b402a1 not found: ID does not exist"
Sep 17 18:41:49.138383 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138293    2566 scope.go:117] "RemoveContainer" containerID="c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"
Sep 17 18:41:49.138476 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.138638 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138616    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26"} err="failed to get container status \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": rpc error: code = NotFound desc = could not find container \"c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26\": container with ID starting with c879ad5f6bb37218b51c75b4af8d05ee85649586c1eea95a92d974c191c25b26 not found: ID does not exist"
Sep 17 18:41:49.138738 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138639    2566 scope.go:117] "RemoveContainer" containerID="caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"
Sep 17 18:41:49.139023 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.138994    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd"} err="failed to get container status \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": rpc error: code = NotFound desc = could not find container \"caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd\": container with ID starting with caca3613e3f90777b053997cc4c25560c946de759281cfca86ce21f50237f1fd not found: ID does not exist"
Sep 17 18:41:49.139023 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.139025    2566 scope.go:117] "RemoveContainer" containerID="c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"
Sep 17 18:41:49.139497 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.139465    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2"} err="failed to get container status \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": rpc error: code = NotFound desc = could not find container \"c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2\": container with ID starting with c956f1a15f35d831e23f0ca0237a9d0aac6d55fd7a5c1ff7a87a41db8b44e4d2 not found: ID does not exist"
Sep 17 18:41:49.139497 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.139488    2566 scope.go:117] "RemoveContainer" containerID="186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"
Sep 17 18:41:49.139775 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.139753    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e"} err="failed to get container status \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": rpc error: code = NotFound desc = could not find container \"186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e\": container with ID starting with 186905d5cd5ee00d7b832a53abf85b52c491438cfa94dd2722da610c3c337b2e not found: ID does not exist"
Sep 17 18:41:49.139879 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.139777    2566 scope.go:117] "RemoveContainer" containerID="1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"
Sep 17 18:41:49.140042 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.140025    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc"} err="failed to get container status \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": rpc error: code = NotFound desc = could not find container \"1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc\": container with ID starting with 1e38fc7b17fa312b4e77f30d9fbee883b9c50825943a0c83ebb8c40d5b9338fc not found: ID does not exist"
Sep 17 18:41:49.140116 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.140045    2566 scope.go:117] "RemoveContainer" containerID="97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"
Sep 17 18:41:49.140363 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.140340    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf"} err="failed to get container status \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": rpc error: code = NotFound desc = could not find container \"97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf\": container with ID starting with 97bfb1e9e9aad5c893be10879792ccf91d82fd6db5977b2831edb9e769896bcf not found: ID does not exist"
Sep 17 18:41:49.141526 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.141509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:41:49.142320 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.142305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:41:49.143020 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.142999    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:41:49.143106 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:41:49.143177 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143131    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:41:49.143177 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:41:49.143177 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:41:49.143334 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143268    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k79wc\""
Sep 17 18:41:49.143334 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143324    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:41:49.143624 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143607    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:41:49.143714 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143692    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:41:49.143767 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:41:49.143767 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143732    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:41:49.143877 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.143692    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-k69mb2f61s1b\""
Sep 17 18:41:49.146753 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.146735    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:41:49.151538 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.151498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:41:49.153531 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.153509    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:49.197629 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.197608    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6507bd-e6f0-41de-851f-8e2f7b87acbd" path="/var/lib/kubelet/pods/ac6507bd-e6f0-41de-851f-8e2f7b87acbd/volumes"
Sep 17 18:41:49.224238 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-config\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5bp\" (UniqueName: \"kubernetes.io/projected/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-kube-api-access-gm5bp\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-config-out\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-web-config\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.224967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.224731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.325976 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.325918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-config-out\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.325976 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.325952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.325976 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.325967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-web-config\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.325987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326166 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-config\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5bp\" (UniqueName: \"kubernetes.io/projected/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-kube-api-access-gm5bp\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.326559 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.327052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.327052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.327052 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.326950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.329083 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.329057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.329185 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.329162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-web-config\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.329536 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.329383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-config-out\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.329536 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.329385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.329536 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.329459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.330413 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.329759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.330413 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.330051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.330413 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.330310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.330889 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.330603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-config\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.330889 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.330657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.331000 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.330909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.331120 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.331097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.331178 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.331114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.331974 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.331951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.332654 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.332633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.332733 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.332652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.338068 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.338041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5bp\" (UniqueName: \"kubernetes.io/projected/9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37-kube-api-access-gm5bp\") pod \"prometheus-k8s-0\" (UID: \"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.448703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.448682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:49.735801 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:49.735780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:49.737887 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:41:49.737855    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9bdc87_afdb_4b90_8bf3_7281ec5e9d37.slice/crio-7bd667cbf860a340dbf256b2dc9f71124b1da62ddff286d54464313f65c243e2 WatchSource:0}: Error finding container 7bd667cbf860a340dbf256b2dc9f71124b1da62ddff286d54464313f65c243e2: Status 404 returned error can't find the container with id 7bd667cbf860a340dbf256b2dc9f71124b1da62ddff286d54464313f65c243e2
Sep 17 18:41:50.074740 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:50.074708    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37" containerID="88eefb66f2b970f6422ac0c396079324dff45d5f2bb664897b04494f21993ab6" exitCode=0
Sep 17 18:41:50.074900 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:50.074802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerDied","Data":"88eefb66f2b970f6422ac0c396079324dff45d5f2bb664897b04494f21993ab6"}
Sep 17 18:41:50.074900 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:50.074897    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"7bd667cbf860a340dbf256b2dc9f71124b1da62ddff286d54464313f65c243e2"}
Sep 17 18:41:51.085241 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.085205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"1da08b201b56b3359f2da65ce8bb02447733fd947823900585f0d0595cc49d02"}
Sep 17 18:41:51.085646 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.085330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"f811bd03e2af8281445c8ff82134906f9c79a61bf28e6abe1ba2c50748afa5ab"}
Sep 17 18:41:51.085646 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.085359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"5ed262c79dd42082adf4f81eb61b82688d287c3a600d0261cb3aa99d49c67157"}
Sep 17 18:41:51.085646 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.085386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"5e0005ba2551c2d2279e588c0dac7fdf193421b343aedf4453c527d97c26a13a"}
Sep 17 18:41:51.085646 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.085410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"b66fc21551c811845d8655e21099b091d170e498d9b02334dc5df5b141e7a3e7"}
Sep 17 18:41:51.085646 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.085433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37","Type":"ContainerStarted","Data":"b684c02d86baea0cb9b5d50cfd5d6271c2e5490d3db294b2d17db066844da09b"}
Sep 17 18:41:51.127799 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:51.127745    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.106564873 podStartE2EDuration="2.127727542s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.162669ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:51.126964457 +0000 UTC m=+187.979668701" watchObservedRunningTime="2026-09-17 18:41:51.127727542 +0000 UTC m=+187.980431787"
Sep 17 18:41:54.449272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:41:54.449229    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:42:49.449728 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:42:49.449676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:42:49.467092 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:42:49.467065    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:42:50.339086 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:42:50.339059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:43:45.092579 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:43:45.092547    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:43:45.100942 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:43:45.100917    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:43:45.103065 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:43:45.103043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:48:45.143099 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:48:45.143066    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:48:45.148247 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:48:45.148226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:50:41.452031 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.449371    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-s55th"]
Sep 17 18:50:41.453845 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.453779    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s55th"
Sep 17 18:50:41.461463 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.461446    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:50:41.462846 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.462808    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9qh76\""
Sep 17 18:50:41.462982 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.462958    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:50:41.463770 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.463751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:50:41.463907 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.463859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:50:41.465355 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.465336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-s55th"]
Sep 17 18:50:41.518251 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.518227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstmj\" (UniqueName: \"kubernetes.io/projected/5876b4df-6114-4f83-88a9-5d6afbcb440f-kube-api-access-hstmj\") pod \"s3-init-s55th\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") " pod="kserve/s3-init-s55th"
Sep 17 18:50:41.518344 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.518279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876b4df-6114-4f83-88a9-5d6afbcb440f-model-cache\") pod \"s3-init-s55th\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") " pod="kserve/s3-init-s55th"
Sep 17 18:50:41.618676 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.618654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876b4df-6114-4f83-88a9-5d6afbcb440f-model-cache\") pod \"s3-init-s55th\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") " pod="kserve/s3-init-s55th"
Sep 17 18:50:41.618764 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.618694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstmj\" (UniqueName: \"kubernetes.io/projected/5876b4df-6114-4f83-88a9-5d6afbcb440f-kube-api-access-hstmj\") pod \"s3-init-s55th\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") " pod="kserve/s3-init-s55th"
Sep 17 18:50:41.619009 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.618993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876b4df-6114-4f83-88a9-5d6afbcb440f-model-cache\") pod \"s3-init-s55th\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") " pod="kserve/s3-init-s55th"
Sep 17 18:50:41.627305 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.627284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstmj\" (UniqueName: \"kubernetes.io/projected/5876b4df-6114-4f83-88a9-5d6afbcb440f-kube-api-access-hstmj\") pod \"s3-init-s55th\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") " pod="kserve/s3-init-s55th"
Sep 17 18:50:41.774566 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:41.774518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s55th"
Sep 17 18:50:42.049129 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:42.049104    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-s55th"]
Sep 17 18:50:42.051136 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:50:42.051109    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5876b4df_6114_4f83_88a9_5d6afbcb440f.slice/crio-9a5bf9ce670852229c251f29318f3d2749fc3fd19c85fef1ce5eda667b4c88be WatchSource:0}: Error finding container 9a5bf9ce670852229c251f29318f3d2749fc3fd19c85fef1ce5eda667b4c88be: Status 404 returned error can't find the container with id 9a5bf9ce670852229c251f29318f3d2749fc3fd19c85fef1ce5eda667b4c88be
Sep 17 18:50:42.052968 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:42.052950    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:50:42.213826 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:42.213798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s55th" event={"ID":"5876b4df-6114-4f83-88a9-5d6afbcb440f","Type":"ContainerStarted","Data":"9a5bf9ce670852229c251f29318f3d2749fc3fd19c85fef1ce5eda667b4c88be"}
Sep 17 18:50:45.413098 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:45.413079    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:50:46.239265 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:46.239227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s55th" event={"ID":"5876b4df-6114-4f83-88a9-5d6afbcb440f","Type":"ContainerStarted","Data":"096cb7abb2c2ffbb29745c5b727149a3712b81519db9d469758caee0a6e801a1"}
Sep 17 18:50:48.249501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:48.249428    2566 generic.go:356] "Generic (PLEG): container finished" podID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerID="096cb7abb2c2ffbb29745c5b727149a3712b81519db9d469758caee0a6e801a1" exitCode=0
Sep 17 18:50:48.249501 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:48.249492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s55th" event={"ID":"5876b4df-6114-4f83-88a9-5d6afbcb440f","Type":"ContainerDied","Data":"096cb7abb2c2ffbb29745c5b727149a3712b81519db9d469758caee0a6e801a1"}
Sep 17 18:50:52.616511 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:52.616483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:50:53.274811 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:53.274778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s55th" event={"ID":"5876b4df-6114-4f83-88a9-5d6afbcb440f","Type":"ContainerStarted","Data":"3056afe742a78b8c86b8b3dbc26a63d7985c8b1135c6b8cde44e7e8d074b6111"}
Sep 17 18:50:53.295205 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:50:53.295152    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-s55th" podStartSLOduration=2.204452452 podStartE2EDuration="12.29513458s" totalImagesPullingTime="7.712020792s" totalInitContainerRuntime="2.378661336s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:53.294053837 +0000 UTC m=+730.146758081" watchObservedRunningTime="2026-09-17 18:50:53.29513458 +0000 UTC m=+730.147838827"
Sep 17 18:51:58.529363 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:58.529328    2566 generic.go:356] "Generic (PLEG): container finished" podID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerID="3056afe742a78b8c86b8b3dbc26a63d7985c8b1135c6b8cde44e7e8d074b6111" exitCode=0
Sep 17 18:51:58.529762 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:58.529402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s55th" event={"ID":"5876b4df-6114-4f83-88a9-5d6afbcb440f","Type":"ContainerDied","Data":"3056afe742a78b8c86b8b3dbc26a63d7985c8b1135c6b8cde44e7e8d074b6111"}
Sep 17 18:51:59.657001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.656978    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s55th"
Sep 17 18:51:59.778239 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.778216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hstmj\" (UniqueName: \"kubernetes.io/projected/5876b4df-6114-4f83-88a9-5d6afbcb440f-kube-api-access-hstmj\") pod \"5876b4df-6114-4f83-88a9-5d6afbcb440f\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") "
Sep 17 18:51:59.778358 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.778246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876b4df-6114-4f83-88a9-5d6afbcb440f-model-cache\") pod \"5876b4df-6114-4f83-88a9-5d6afbcb440f\" (UID: \"5876b4df-6114-4f83-88a9-5d6afbcb440f\") "
Sep 17 18:51:59.780272 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.780245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5876b4df-6114-4f83-88a9-5d6afbcb440f-kube-api-access-hstmj" (OuterVolumeSpecName: "kube-api-access-hstmj") pod "5876b4df-6114-4f83-88a9-5d6afbcb440f" (UID: "5876b4df-6114-4f83-88a9-5d6afbcb440f"). InnerVolumeSpecName "kube-api-access-hstmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:59.797041 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.797016    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5876b4df-6114-4f83-88a9-5d6afbcb440f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5876b4df-6114-4f83-88a9-5d6afbcb440f" (UID: "5876b4df-6114-4f83-88a9-5d6afbcb440f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:51:59.879711 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.879689    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstmj\" (UniqueName: \"kubernetes.io/projected/5876b4df-6114-4f83-88a9-5d6afbcb440f-kube-api-access-hstmj\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:51:59.879711 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:51:59.879708    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876b4df-6114-4f83-88a9-5d6afbcb440f-model-cache\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.538648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:00.538606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s55th" event={"ID":"5876b4df-6114-4f83-88a9-5d6afbcb440f","Type":"ContainerDied","Data":"9a5bf9ce670852229c251f29318f3d2749fc3fd19c85fef1ce5eda667b4c88be"}
Sep 17 18:52:00.538648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:00.538648    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5bf9ce670852229c251f29318f3d2749fc3fd19c85fef1ce5eda667b4c88be"
Sep 17 18:52:00.538648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:00.538638    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s55th"
Sep 17 18:52:12.589131 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.589097    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"]
Sep 17 18:52:12.589575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.589554    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerName="download-hf-model"
Sep 17 18:52:12.589575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.589574    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerName="download-hf-model"
Sep 17 18:52:12.589700 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.589586    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerName="s3-init"
Sep 17 18:52:12.589700 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.589594    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerName="s3-init"
Sep 17 18:52:12.589700 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.589671    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5876b4df-6114-4f83-88a9-5d6afbcb440f" containerName="s3-init"
Sep 17 18:52:12.592963 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.592940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.596661 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.596639    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:52:12.596816 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.596672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:52:12.596816 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.596707    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:52:12.597114 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.597095    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:52:12.597716 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.597698    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:52:12.597810 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.597746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 18:52:12.597901 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.597810    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 18:52:12.604242 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.604223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"]
Sep 17 18:52:12.673736 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.673715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.673873 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.673745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmhw\" (UniqueName: \"kubernetes.io/projected/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-api-access-xfmhw\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.673873 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.673763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.673873 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.673781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.673993 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.673879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.705621 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.705598    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"]
Sep 17 18:52:12.708988 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.708973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.711225 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.711209    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:52:12.720067 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.720045    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"]
Sep 17 18:52:12.775003 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.774982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnppr\" (UniqueName: \"kubernetes.io/projected/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-api-access-hnppr\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.775110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.775110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmhw\" (UniqueName: \"kubernetes.io/projected/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-api-access-xfmhw\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775110 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775345 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.775345 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775345 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.775345 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.775549 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775796 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.775882 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.775781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.777512 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.777494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.785170 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.785152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmhw\" (UniqueName: \"kubernetes.io/projected/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-api-access-xfmhw\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:12.875900 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.875851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.875900 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.875881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.876063 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.875899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnppr\" (UniqueName: \"kubernetes.io/projected/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-api-access-hnppr\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.876063 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.876031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.876170 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.876106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.876349 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.876326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.876565 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.876544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.876626 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.876609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.878040 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.878023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.885546 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.885527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnppr\" (UniqueName: \"kubernetes.io/projected/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-api-access-hnppr\") pod \"isvc-canary-rollback-predictor-69978998f6-pcnqt\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:12.903908 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:12.903889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:13.020643 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.020618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:13.177858 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.177819    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"]
Sep 17 18:52:13.179312 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:52:13.179286    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db0e2cb_6c2b_4a71_9685_75e67d08ec20.slice/crio-a02b2d512e6f15a8816c434f333f7a280ba5e95bedfff8c44652cc185f69f542 WatchSource:0}: Error finding container a02b2d512e6f15a8816c434f333f7a280ba5e95bedfff8c44652cc185f69f542: Status 404 returned error can't find the container with id a02b2d512e6f15a8816c434f333f7a280ba5e95bedfff8c44652cc185f69f542
Sep 17 18:52:13.296359 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.296335    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"]
Sep 17 18:52:13.299306 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:52:13.299279    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ebc095_6c7e_4cf8_8fd6_8b73a135f6ce.slice/crio-948d0e94e26ea9943a59e72384b69c3c388b92e9c493b8c177b4aefe3ca15bea WatchSource:0}: Error finding container 948d0e94e26ea9943a59e72384b69c3c388b92e9c493b8c177b4aefe3ca15bea: Status 404 returned error can't find the container with id 948d0e94e26ea9943a59e72384b69c3c388b92e9c493b8c177b4aefe3ca15bea
Sep 17 18:52:13.592523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.592489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerStarted","Data":"9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa"}
Sep 17 18:52:13.593030 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.592566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerStarted","Data":"948d0e94e26ea9943a59e72384b69c3c388b92e9c493b8c177b4aefe3ca15bea"}
Sep 17 18:52:13.594315 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.594289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerStarted","Data":"6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837"}
Sep 17 18:52:13.594450 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:13.594335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerStarted","Data":"a02b2d512e6f15a8816c434f333f7a280ba5e95bedfff8c44652cc185f69f542"}
Sep 17 18:52:17.611533 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:17.611496    2566 generic.go:356] "Generic (PLEG): container finished" podID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerID="6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837" exitCode=0
Sep 17 18:52:17.611994 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:17.611568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerDied","Data":"6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837"}
Sep 17 18:52:17.613242 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:17.613209    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerID="9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa" exitCode=0
Sep 17 18:52:17.613343 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:17.613283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerDied","Data":"9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa"}
Sep 17 18:52:34.726444 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:34.726406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerStarted","Data":"7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be"}
Sep 17 18:52:34.729056 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:34.729018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerStarted","Data":"06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05"}
Sep 17 18:52:37.751888 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:37.751843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerStarted","Data":"00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b"}
Sep 17 18:52:37.754333 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:37.754305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerStarted","Data":"c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae"}
Sep 17 18:52:37.754456 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:37.754430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:37.772526 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:37.772482    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podStartSLOduration=2.585006339 podStartE2EDuration="25.772468744s" totalImagesPullingTime="19.123991103s" totalInitContainerRuntime="4.063471302s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:37.771254758 +0000 UTC m=+834.623958998" watchObservedRunningTime="2026-09-17 18:52:37.772468744 +0000 UTC m=+834.625173058"
Sep 17 18:52:37.790876 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:37.790817    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podStartSLOduration=2.9364236999999997 podStartE2EDuration="25.790800593s" totalImagesPullingTime="19.177453821s" totalInitContainerRuntime="3.676923072s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:37.789154818 +0000 UTC m=+834.641859061" watchObservedRunningTime="2026-09-17 18:52:37.790800593 +0000 UTC m=+834.643504839"
Sep 17 18:52:38.759004 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:38.758974    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:38.759419 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:38.759038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:38.759419 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:38.759074    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:38.759870 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:38.759815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:38.760020 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:38.759995    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:39.762486 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:39.762444    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:39.762956 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:39.762621    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:44.768100 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:44.768071    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:52:44.768553 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:44.768180    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:52:44.768665 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:44.768641    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:44.768737 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:44.768681    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:54.769540 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:54.769495    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:54.769956 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:52:54.769561    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:53:04.768741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:04.768696    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:04.769145 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:04.768696    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:53:14.768716 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:14.768679    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:53:14.769177 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:14.768681    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:24.768914 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:24.768873    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:24.769268 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:24.768952    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:53:34.768998 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:34.768910    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:53:34.769520 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:34.768907    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:44.768694 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:44.768653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:44.768694 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:44.768653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:53:45.176369 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:45.176340    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:53:45.178093 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:45.178072    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:53:54.769653 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:54.769617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:53:54.770097 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:53:54.769857    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:54:02.498046 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:02.498015    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"]
Sep 17 18:54:02.498466 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:02.498256    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" containerID="cri-o://7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be" gracePeriod=30
Sep 17 18:54:02.498466 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:02.498314    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kube-rbac-proxy" containerID="cri-o://00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b" gracePeriod=30
Sep 17 18:54:03.118746 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:03.118715    2566 generic.go:356] "Generic (PLEG): container finished" podID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerID="00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b" exitCode=2
Sep 17 18:54:03.118916 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:03.118788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerDied","Data":"00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b"}
Sep 17 18:54:04.763064 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:04.763020    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 18:54:04.769482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:04.769446    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:54:06.632908 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.632882    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:54:06.656457 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656432    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfmhw\" (UniqueName: \"kubernetes.io/projected/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-api-access-xfmhw\") pod \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") "
Sep 17 18:54:06.656575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-cabundle-cert\") pod \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") "
Sep 17 18:54:06.656575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656499    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kserve-provision-location\") pod \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") "
Sep 17 18:54:06.656575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-rbac-proxy-sar-config\") pod \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") "
Sep 17 18:54:06.656575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656565    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-proxy-tls\") pod \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\" (UID: \"0db0e2cb-6c2b-4a71-9685-75e67d08ec20\") "
Sep 17 18:54:06.656842 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0db0e2cb-6c2b-4a71-9685-75e67d08ec20" (UID: "0db0e2cb-6c2b-4a71-9685-75e67d08ec20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:06.656954 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656813    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0db0e2cb-6c2b-4a71-9685-75e67d08ec20" (UID: "0db0e2cb-6c2b-4a71-9685-75e67d08ec20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:06.656954 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.656926    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0db0e2cb-6c2b-4a71-9685-75e67d08ec20" (UID: "0db0e2cb-6c2b-4a71-9685-75e67d08ec20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:06.658753 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.658724    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0db0e2cb-6c2b-4a71-9685-75e67d08ec20" (UID: "0db0e2cb-6c2b-4a71-9685-75e67d08ec20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:06.658966 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.658947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-api-access-xfmhw" (OuterVolumeSpecName: "kube-api-access-xfmhw") pod "0db0e2cb-6c2b-4a71-9685-75e67d08ec20" (UID: "0db0e2cb-6c2b-4a71-9685-75e67d08ec20"). InnerVolumeSpecName "kube-api-access-xfmhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:06.757794 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.757744    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmhw\" (UniqueName: \"kubernetes.io/projected/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-api-access-xfmhw\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.757794 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.757766    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.757794 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.757788    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.757794 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.757796    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.757993 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:06.757806    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db0e2cb-6c2b-4a71-9685-75e67d08ec20-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:54:07.139343 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.139314    2566 generic.go:356] "Generic (PLEG): container finished" podID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerID="7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be" exitCode=0
Sep 17 18:54:07.139490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.139374    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"
Sep 17 18:54:07.139490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.139394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerDied","Data":"7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be"}
Sep 17 18:54:07.139490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.139444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt" event={"ID":"0db0e2cb-6c2b-4a71-9685-75e67d08ec20","Type":"ContainerDied","Data":"a02b2d512e6f15a8816c434f333f7a280ba5e95bedfff8c44652cc185f69f542"}
Sep 17 18:54:07.139490 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.139471    2566 scope.go:117] "RemoveContainer" containerID="00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b"
Sep 17 18:54:07.149054 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.149038    2566 scope.go:117] "RemoveContainer" containerID="7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be"
Sep 17 18:54:07.156528 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.156509    2566 scope.go:117] "RemoveContainer" containerID="6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837"
Sep 17 18:54:07.163819 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.163802    2566 scope.go:117] "RemoveContainer" containerID="00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b"
Sep 17 18:54:07.164418 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:07.164281    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b\": container with ID starting with 00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b not found: ID does not exist" containerID="00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b"
Sep 17 18:54:07.164418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.164316    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b"} err="failed to get container status \"00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b\": rpc error: code = NotFound desc = could not find container \"00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b\": container with ID starting with 00ddbc22c92f45381b910ee72a8502432a49e44cfd7522ccf43b2d872edbc88b not found: ID does not exist"
Sep 17 18:54:07.164418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.164339    2566 scope.go:117] "RemoveContainer" containerID="7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be"
Sep 17 18:54:07.165168 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:07.164819    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be\": container with ID starting with 7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be not found: ID does not exist" containerID="7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be"
Sep 17 18:54:07.165262 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.165175    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be"} err="failed to get container status \"7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be\": rpc error: code = NotFound desc = could not find container \"7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be\": container with ID starting with 7965b10ed2d61812bb8dd031d309c481c08e1419d8407b6603a6e386e38ce1be not found: ID does not exist"
Sep 17 18:54:07.165262 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.165191    2566 scope.go:117] "RemoveContainer" containerID="6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837"
Sep 17 18:54:07.165488 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:07.165473    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837\": container with ID starting with 6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837 not found: ID does not exist" containerID="6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837"
Sep 17 18:54:07.165538 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.165490    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837"} err="failed to get container status \"6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837\": rpc error: code = NotFound desc = could not find container \"6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837\": container with ID starting with 6d84bbaf6a16cba0a0b418551e8d5b1547a6bb65c34cbbb98c480aeadb3ba837 not found: ID does not exist"
Sep 17 18:54:07.166007 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.165988    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"]
Sep 17 18:54:07.172644 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.172622    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-zk6zt"]
Sep 17 18:54:07.198018 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.197998    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" path="/var/lib/kubelet/pods/0db0e2cb-6c2b-4a71-9685-75e67d08ec20/volumes"
Sep 17 18:54:07.792813 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.792781    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"]
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793085    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793097    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793113    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="storage-initializer"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793119    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="storage-initializer"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793132    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kube-rbac-proxy"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kube-rbac-proxy"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793202    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kserve-container"
Sep 17 18:54:07.793209 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.793211    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db0e2cb-6c2b-4a71-9685-75e67d08ec20" containerName="kube-rbac-proxy"
Sep 17 18:54:07.797615 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.797599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.799895 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.799871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-bae88-predictor-serving-cert\""
Sep 17 18:54:07.799895 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.799870    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-bae88-kube-rbac-proxy-sar-config\""
Sep 17 18:54:07.810810 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.810790    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"]
Sep 17 18:54:07.864007 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.863979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.864127 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.864017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d24367-4972-4b7b-b6ac-8340d427c873-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.864127 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.864052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2627\" (UniqueName: \"kubernetes.io/projected/40d24367-4972-4b7b-b6ac-8340d427c873-kube-api-access-z2627\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.864127 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.864082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.864244 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.864126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-cabundle-cert\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.910640 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.910619    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"]
Sep 17 18:54:07.914112 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.914094    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:07.916914 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.916894    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-bae88-predictor-serving-cert\""
Sep 17 18:54:07.917567 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.917548    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-bae88-kube-rbac-proxy-sar-config\""
Sep 17 18:54:07.927732 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.927714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"]
Sep 17 18:54:07.965461 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:07.965551 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2627\" (UniqueName: \"kubernetes.io/projected/40d24367-4972-4b7b-b6ac-8340d427c873-kube-api-access-z2627\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.965551 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:07.965551 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.965651 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2nb\" (UniqueName: \"kubernetes.io/projected/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-api-access-pt2nb\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:07.965651 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:07.965599    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-serving-cert: secret "isvc-sklearn-graph-raw-bae88-predictor-serving-cert" not found
Sep 17 18:54:07.965651 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-cabundle-cert\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.965741 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:07.965654    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls podName:40d24367-4972-4b7b-b6ac-8340d427c873 nodeName:}" failed. No retries permitted until 2026-09-17 18:54:08.465634946 +0000 UTC m=+925.318339168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls") pod "isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" (UID: "40d24367-4972-4b7b-b6ac-8340d427c873") : secret "isvc-sklearn-graph-raw-bae88-predictor-serving-cert" not found
Sep 17 18:54:07.965741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:07.965741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-cabundle-cert\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:07.965741 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.965938 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.965785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d24367-4972-4b7b-b6ac-8340d427c873-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.966095 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.966076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d24367-4972-4b7b-b6ac-8340d427c873-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.966233 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.966217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-cabundle-cert\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.966295 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.966279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:07.976538 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:07.976513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2627\" (UniqueName: \"kubernetes.io/projected/40d24367-4972-4b7b-b6ac-8340d427c873-kube-api-access-z2627\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:08.066381 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.066360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.066489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.066404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2nb\" (UniqueName: \"kubernetes.io/projected/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-api-access-pt2nb\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.066489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.066442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.066489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.066457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-cabundle-cert\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.066489 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.066485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.066719 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:08.066620    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-serving-cert: secret "isvc-xgboost-graph-raw-bae88-predictor-serving-cert" not found
Sep 17 18:54:08.066719 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:54:08.066681    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls podName:d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31 nodeName:}" failed. No retries permitted until 2026-09-17 18:54:08.566665856 +0000 UTC m=+925.419370078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls") pod "isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" (UID: "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31") : secret "isvc-xgboost-graph-raw-bae88-predictor-serving-cert" not found
Sep 17 18:54:08.066860 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.066789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.067137 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.067119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.067174 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.067158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-cabundle-cert\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.078308 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.078284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2nb\" (UniqueName: \"kubernetes.io/projected/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-api-access-pt2nb\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.470088 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.470023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:08.472321 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.472301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls\") pod \"isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:08.570522 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.570491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.572611 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.572593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls\") pod \"isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.707553 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.707529    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:08.825383 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.825351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:08.985973 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:08.981375    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"]
Sep 17 18:54:08.989811 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:54:08.989786    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d24367_4972_4b7b_b6ac_8340d427c873.slice/crio-5d82c4ac36434fcf3bf9aaa5f44eb48fe8e0eee20948e310896ad1c6d70b9d86 WatchSource:0}: Error finding container 5d82c4ac36434fcf3bf9aaa5f44eb48fe8e0eee20948e310896ad1c6d70b9d86: Status 404 returned error can't find the container with id 5d82c4ac36434fcf3bf9aaa5f44eb48fe8e0eee20948e310896ad1c6d70b9d86
Sep 17 18:54:09.102470 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:09.102443    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"]
Sep 17 18:54:09.106286 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:54:09.106253    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73fbbd0_2aeb_4cd8_9b68_1b53c66c6d31.slice/crio-3db3e14d3c6c7c7e3ad17f79504bacedb9d0d6e114be571515a5b2033986b7fb WatchSource:0}: Error finding container 3db3e14d3c6c7c7e3ad17f79504bacedb9d0d6e114be571515a5b2033986b7fb: Status 404 returned error can't find the container with id 3db3e14d3c6c7c7e3ad17f79504bacedb9d0d6e114be571515a5b2033986b7fb
Sep 17 18:54:09.151027 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:09.150998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerStarted","Data":"3db3e14d3c6c7c7e3ad17f79504bacedb9d0d6e114be571515a5b2033986b7fb"}
Sep 17 18:54:09.152702 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:09.152647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerStarted","Data":"1d95bc161fefe27ebc75fada1812231b1601ad03663fa23427d1deaa31f7ca92"}
Sep 17 18:54:09.152815 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:09.152711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerStarted","Data":"5d82c4ac36434fcf3bf9aaa5f44eb48fe8e0eee20948e310896ad1c6d70b9d86"}
Sep 17 18:54:10.158880 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:10.158843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerStarted","Data":"15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02"}
Sep 17 18:54:13.174167 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:13.174139    2566 generic.go:356] "Generic (PLEG): container finished" podID="40d24367-4972-4b7b-b6ac-8340d427c873" containerID="1d95bc161fefe27ebc75fada1812231b1601ad03663fa23427d1deaa31f7ca92" exitCode=0
Sep 17 18:54:13.174525 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:13.174214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerDied","Data":"1d95bc161fefe27ebc75fada1812231b1601ad03663fa23427d1deaa31f7ca92"}
Sep 17 18:54:13.175997 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:13.175975    2566 generic.go:356] "Generic (PLEG): container finished" podID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerID="15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02" exitCode=0
Sep 17 18:54:13.176071 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:13.176026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerDied","Data":"15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02"}
Sep 17 18:54:14.184481 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:14.184451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerStarted","Data":"861f50c1ad7cce1651ed6d9afde5c092a2e88ec4c141147f893dc710aae51893"}
Sep 17 18:54:14.184952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:14.184528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerStarted","Data":"273f4ede020f8e93be0de1d452149febab0a6f4c64071bb26185cf74a97c9572"}
Sep 17 18:54:14.184952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:14.184816    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:14.184952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:14.184902    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:14.186653 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:14.186617    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:54:14.210565 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:14.210511    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podStartSLOduration=3.532118591 podStartE2EDuration="7.210485006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.678366415s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:14.208959745 +0000 UTC m=+931.061663988" watchObservedRunningTime="2026-09-17 18:54:14.210485006 +0000 UTC m=+931.063189250"
Sep 17 18:54:15.191521 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:15.191382    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:54:20.197608 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:20.197579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:54:20.198170 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:20.198143    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:54:30.198350 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:30.198305    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:54:32.281298 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:32.281227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerStarted","Data":"67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb"}
Sep 17 18:54:32.281298 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:32.281285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerStarted","Data":"f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585"}
Sep 17 18:54:32.281704 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:32.281453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:32.303395 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:32.303346    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podStartSLOduration=2.691510915 podStartE2EDuration="25.303330561s" totalImagesPullingTime="18.826737174s" totalInitContainerRuntime="3.785082472s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:32.302086481 +0000 UTC m=+949.154790722" watchObservedRunningTime="2026-09-17 18:54:32.303330561 +0000 UTC m=+949.156034805"
Sep 17 18:54:33.285714 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:33.285681    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:33.286878 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:33.286852    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:54:34.290250 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:34.290210    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:54:39.295056 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:39.295030    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:54:39.295534 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:39.295509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:54:40.198737 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:40.198701    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:54:49.295547 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:49.295506    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:54:50.198104 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:50.198070    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:54:59.295986 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:54:59.295904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:55:00.198321 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:00.198284    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:55:09.295420 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:09.295375    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:55:10.198589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:10.198551    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:55:19.295467 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:19.295420    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:55:20.198418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:20.198378    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:55:24.193664 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:24.193621    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:55:29.295612 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:29.295572    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:55:32.640346 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:32.640316    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"]
Sep 17 18:55:32.640711 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:32.640557    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" containerID="cri-o://06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05" gracePeriod=30
Sep 17 18:55:32.640711 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:32.640609    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kube-rbac-proxy" containerID="cri-o://c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae" gracePeriod=30
Sep 17 18:55:33.544764 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:33.544728    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerID="c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae" exitCode=2
Sep 17 18:55:33.544941 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:33.544768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerDied","Data":"c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae"}
Sep 17 18:55:34.194039 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:34.194003    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:55:34.762980 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:34.762935    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 18:55:34.769251 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:34.769223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:55:36.876418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.876397    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:55:36.971500 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnppr\" (UniqueName: \"kubernetes.io/projected/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-api-access-hnppr\") pod \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") "
Sep 17 18:55:36.971500 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-proxy-tls\") pod \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") "
Sep 17 18:55:36.971689 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-rbac-proxy-sar-config\") pod \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") "
Sep 17 18:55:36.971689 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971526    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kserve-provision-location\") pod \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") "
Sep 17 18:55:36.971689 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-cabundle-cert\") pod \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\" (UID: \"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce\") "
Sep 17 18:55:36.971888 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" (UID: "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:55:36.971952 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.971872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" (UID: "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:36.972065 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.972038    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" (UID: "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:55:36.973433 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.973410    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-api-access-hnppr" (OuterVolumeSpecName: "kube-api-access-hnppr") pod "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" (UID: "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce"). InnerVolumeSpecName "kube-api-access-hnppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:55:36.973527 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:36.973508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" (UID: "f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:55:37.072993 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.072967    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:55:37.072993 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.072992    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnppr\" (UniqueName: \"kubernetes.io/projected/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-api-access-hnppr\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:55:37.073126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.073006    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:55:37.073126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.073022    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:55:37.073126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.073035    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:55:37.565515 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.565487    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerID="06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05" exitCode=0
Sep 17 18:55:37.565648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.565539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerDied","Data":"06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05"}
Sep 17 18:55:37.565648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.565551    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"
Sep 17 18:55:37.565648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.565594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt" event={"ID":"f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce","Type":"ContainerDied","Data":"948d0e94e26ea9943a59e72384b69c3c388b92e9c493b8c177b4aefe3ca15bea"}
Sep 17 18:55:37.565648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.565623    2566 scope.go:117] "RemoveContainer" containerID="c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae"
Sep 17 18:55:37.574702 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.574682    2566 scope.go:117] "RemoveContainer" containerID="06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05"
Sep 17 18:55:37.582438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.582413    2566 scope.go:117] "RemoveContainer" containerID="9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa"
Sep 17 18:55:37.585583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.585561    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"]
Sep 17 18:55:37.588791 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.588769    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-pcnqt"]
Sep 17 18:55:37.590980 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.590952    2566 scope.go:117] "RemoveContainer" containerID="c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae"
Sep 17 18:55:37.591240 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:55:37.591222    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae\": container with ID starting with c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae not found: ID does not exist" containerID="c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae"
Sep 17 18:55:37.591309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.591250    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae"} err="failed to get container status \"c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae\": rpc error: code = NotFound desc = could not find container \"c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae\": container with ID starting with c951a4b27e2f364057b7214a0692a21209944067be4de3928292742ff33ef6ae not found: ID does not exist"
Sep 17 18:55:37.591309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.591268    2566 scope.go:117] "RemoveContainer" containerID="06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05"
Sep 17 18:55:37.591533 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:55:37.591514    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05\": container with ID starting with 06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05 not found: ID does not exist" containerID="06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05"
Sep 17 18:55:37.591596 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.591543    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05"} err="failed to get container status \"06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05\": rpc error: code = NotFound desc = could not find container \"06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05\": container with ID starting with 06e8e4def6e48113fc465df6d68871d8c887d889cac5f2c4bb652f06d0a57e05 not found: ID does not exist"
Sep 17 18:55:37.591596 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.591566    2566 scope.go:117] "RemoveContainer" containerID="9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa"
Sep 17 18:55:37.591872 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:55:37.591848    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa\": container with ID starting with 9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa not found: ID does not exist" containerID="9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa"
Sep 17 18:55:37.591976 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:37.591876    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa"} err="failed to get container status \"9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa\": rpc error: code = NotFound desc = could not find container \"9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa\": container with ID starting with 9f6f9a3a6f65de76273cd4edc82339dfb27c4dbdfb9d75c77491ca38753e13fa not found: ID does not exist"
Sep 17 18:55:38.034329 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034219    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"]
Sep 17 18:55:38.034755 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034679    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kube-rbac-proxy"
Sep 17 18:55:38.034755 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034698    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kube-rbac-proxy"
Sep 17 18:55:38.034755 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="storage-initializer"
Sep 17 18:55:38.034755 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034731    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="storage-initializer"
Sep 17 18:55:38.035001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034761    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container"
Sep 17 18:55:38.035001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034770    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container"
Sep 17 18:55:38.035001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034861    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kserve-container"
Sep 17 18:55:38.035001 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.034875    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" containerName="kube-rbac-proxy"
Sep 17 18:55:38.039452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.039431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.042538 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.042517    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:55:38.042747 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.042732    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:55:38.056583 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.056560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"]
Sep 17 18:55:38.180205 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.180183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.180309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.180226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw7m\" (UniqueName: \"kubernetes.io/projected/30c0b269-160d-4534-b4a3-c51a5929e516-kube-api-access-4sw7m\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.180309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.180248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c0b269-160d-4534-b4a3-c51a5929e516-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.180309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.180272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.180309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.180291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281071 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw7m\" (UniqueName: \"kubernetes.io/projected/30c0b269-160d-4534-b4a3-c51a5929e516-kube-api-access-4sw7m\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281183 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c0b269-160d-4534-b4a3-c51a5929e516-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281183 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281183 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281183 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281387 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:55:38.281262    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:55:38.281387 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:55:38.281334    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls podName:30c0b269-160d-4534-b4a3-c51a5929e516 nodeName:}" failed. No retries permitted until 2026-09-17 18:55:38.781313883 +0000 UTC m=+1015.634018126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls") pod "isvc-canary-stop-predictor-6958d6c864-4srhf" (UID: "30c0b269-160d-4534-b4a3-c51a5929e516") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:55:38.281540 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c0b269-160d-4534-b4a3-c51a5929e516-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281732 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.281804 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.281788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.290651 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.290598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw7m\" (UniqueName: \"kubernetes.io/projected/30c0b269-160d-4534-b4a3-c51a5929e516-kube-api-access-4sw7m\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.784452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.784421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.786922 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.786898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-4srhf\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:38.949117 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:38.949092    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:39.199293 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:39.199263    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce" path="/var/lib/kubelet/pods/f2ebc095-6c7e-4cf8-8fd6-8b73a135f6ce/volumes"
Sep 17 18:55:39.226103 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:39.226081    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"]
Sep 17 18:55:39.228131 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:55:39.228103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c0b269_160d_4534_b4a3_c51a5929e516.slice/crio-cfead21f3bc6b8be31475d14137582cab69ab55a925c1cf50bfceb2cfa7b1b81 WatchSource:0}: Error finding container cfead21f3bc6b8be31475d14137582cab69ab55a925c1cf50bfceb2cfa7b1b81: Status 404 returned error can't find the container with id cfead21f3bc6b8be31475d14137582cab69ab55a925c1cf50bfceb2cfa7b1b81
Sep 17 18:55:39.296012 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:39.295983    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:55:39.582039 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:39.582005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerStarted","Data":"7a2132fe6c8d0f77b650068ced801ca9139fcb20c718877fe8aaca00f9ef8393"}
Sep 17 18:55:39.582197 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:39.582066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerStarted","Data":"cfead21f3bc6b8be31475d14137582cab69ab55a925c1cf50bfceb2cfa7b1b81"}
Sep 17 18:55:43.602958 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:43.602921    2566 generic.go:356] "Generic (PLEG): container finished" podID="30c0b269-160d-4534-b4a3-c51a5929e516" containerID="7a2132fe6c8d0f77b650068ced801ca9139fcb20c718877fe8aaca00f9ef8393" exitCode=0
Sep 17 18:55:43.603450 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:43.602968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerDied","Data":"7a2132fe6c8d0f77b650068ced801ca9139fcb20c718877fe8aaca00f9ef8393"}
Sep 17 18:55:44.609159 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:44.609124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerStarted","Data":"fe6e8411da32424e5a74448a66c18471127bd0f05c53ffad4bf37e7bc56d615b"}
Sep 17 18:55:44.609525 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:44.609184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerStarted","Data":"c44d0a4dde51bf9eb429d32d1d379adeb53232b03ad36d019a9ca9bf864017eb"}
Sep 17 18:55:44.609525 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:44.609376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:44.631853 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:44.631775    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podStartSLOduration=2.924016205 podStartE2EDuration="6.631761978s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707745773s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:44.629500947 +0000 UTC m=+1021.482205191" watchObservedRunningTime="2026-09-17 18:55:44.631761978 +0000 UTC m=+1021.484466222"
Sep 17 18:55:45.613424 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:45.613398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:45.614669 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:45.614645    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:55:46.618003 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:46.617956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:55:51.622751 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:51.622717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:55:51.623364 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:51.623333    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:55:58.017660 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.017625    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"]
Sep 17 18:55:58.018033 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.017891    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container" containerID="cri-o://273f4ede020f8e93be0de1d452149febab0a6f4c64071bb26185cf74a97c9572" gracePeriod=30
Sep 17 18:55:58.018033 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.017920    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kube-rbac-proxy" containerID="cri-o://861f50c1ad7cce1651ed6d9afde5c092a2e88ec4c141147f893dc710aae51893" gracePeriod=30
Sep 17 18:55:58.077187 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.077157    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"]
Sep 17 18:55:58.077465 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.077441    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" containerID="cri-o://f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585" gracePeriod=30
Sep 17 18:55:58.077552 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.077523    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kube-rbac-proxy" containerID="cri-o://67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb" gracePeriod=30
Sep 17 18:55:58.086910 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:55:58.086885    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73fbbd0_2aeb_4cd8_9b68_1b53c66c6d31.slice/crio-67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:55:58.312869 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.312844    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"]
Sep 17 18:55:58.316555 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.316529    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.319475 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.319455    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-serving-cert\""
Sep 17 18:55:58.319717 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.319696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-8b7b7-kube-rbac-proxy-sar-config\""
Sep 17 18:55:58.336020 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.335998    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"]
Sep 17 18:55:58.341014 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.340975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.341126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.341041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.341126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.341082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.341126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.341116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d0b861-c13d-475f-a8b1-dc8840fa98df-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.341297 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.341188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdtb\" (UniqueName: \"kubernetes.io/projected/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-api-access-7wdtb\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442069 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442244 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442244 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442244 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d0b861-c13d-475f-a8b1-dc8840fa98df-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442413 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdtb\" (UniqueName: \"kubernetes.io/projected/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-api-access-7wdtb\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442875 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.442875 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.443020 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.442932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.445097 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.445066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d0b861-c13d-475f-a8b1-dc8840fa98df-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.455998 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.455973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdtb\" (UniqueName: \"kubernetes.io/projected/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-api-access-7wdtb\") pod \"isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.627453 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.627380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:55:58.673319 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.673286    2566 generic.go:356] "Generic (PLEG): container finished" podID="40d24367-4972-4b7b-b6ac-8340d427c873" containerID="861f50c1ad7cce1651ed6d9afde5c092a2e88ec4c141147f893dc710aae51893" exitCode=2
Sep 17 18:55:58.673482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.673359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerDied","Data":"861f50c1ad7cce1651ed6d9afde5c092a2e88ec4c141147f893dc710aae51893"}
Sep 17 18:55:58.676965 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.676942    2566 generic.go:356] "Generic (PLEG): container finished" podID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerID="67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb" exitCode=2
Sep 17 18:55:58.677081 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.677004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerDied","Data":"67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb"}
Sep 17 18:55:58.905189 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:58.905166    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"]
Sep 17 18:55:58.907063 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:55:58.907035    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d0b861_c13d_475f_a8b1_dc8840fa98df.slice/crio-a40ecf989fc8be9e90f774a5b97d99fbbc91f10d7238d99251c8f19660f08e8a WatchSource:0}: Error finding container a40ecf989fc8be9e90f774a5b97d99fbbc91f10d7238d99251c8f19660f08e8a: Status 404 returned error can't find the container with id a40ecf989fc8be9e90f774a5b97d99fbbc91f10d7238d99251c8f19660f08e8a
Sep 17 18:55:59.290475 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:59.290381    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 18:55:59.295739 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:59.295706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:55:59.683193 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:59.683157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerStarted","Data":"b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30"}
Sep 17 18:55:59.683362 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:55:59.683226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerStarted","Data":"a40ecf989fc8be9e90f774a5b97d99fbbc91f10d7238d99251c8f19660f08e8a"}
Sep 17 18:56:00.192363 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:00.192324    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 18:56:01.623964 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.623925    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:56:01.923197 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.923175    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:56:01.971955 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.971931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-rbac-proxy-sar-config\") pod \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") "
Sep 17 18:56:01.971955 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.971961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-cabundle-cert\") pod \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") "
Sep 17 18:56:01.972165 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.972000    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls\") pod \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") "
Sep 17 18:56:01.972165 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.972025    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt2nb\" (UniqueName: \"kubernetes.io/projected/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-api-access-pt2nb\") pod \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") "
Sep 17 18:56:01.972165 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.972092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kserve-provision-location\") pod \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\" (UID: \"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31\") "
Sep 17 18:56:01.972347 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.972318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" (UID: "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:01.972459 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.972415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" (UID: "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:01.972556 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.972475    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" (UID: "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.974175 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.974154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-api-access-pt2nb" (OuterVolumeSpecName: "kube-api-access-pt2nb") pod "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" (UID: "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31"). InnerVolumeSpecName "kube-api-access-pt2nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:01.974253 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:01.974175    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" (UID: "d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:02.073012 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.072986    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.073012 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.073007    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.073132 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.073018    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.073132 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.073029    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.073132 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.073037    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2nb\" (UniqueName: \"kubernetes.io/projected/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31-kube-api-access-pt2nb\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.700006 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.699970    2566 generic.go:356] "Generic (PLEG): container finished" podID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerID="f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585" exitCode=0
Sep 17 18:56:02.700523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.700127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerDied","Data":"f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585"}
Sep 17 18:56:02.700523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.700140    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"
Sep 17 18:56:02.700523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.700194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57" event={"ID":"d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31","Type":"ContainerDied","Data":"3db3e14d3c6c7c7e3ad17f79504bacedb9d0d6e114be571515a5b2033986b7fb"}
Sep 17 18:56:02.700523 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.700231    2566 scope.go:117] "RemoveContainer" containerID="67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb"
Sep 17 18:56:02.704055 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.704034    2566 generic.go:356] "Generic (PLEG): container finished" podID="40d24367-4972-4b7b-b6ac-8340d427c873" containerID="273f4ede020f8e93be0de1d452149febab0a6f4c64071bb26185cf74a97c9572" exitCode=0
Sep 17 18:56:02.704172 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.704097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerDied","Data":"273f4ede020f8e93be0de1d452149febab0a6f4c64071bb26185cf74a97c9572"}
Sep 17 18:56:02.715343 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.715321    2566 scope.go:117] "RemoveContainer" containerID="f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585"
Sep 17 18:56:02.723063 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.723040    2566 scope.go:117] "RemoveContainer" containerID="15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02"
Sep 17 18:56:02.731648 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.731628    2566 scope.go:117] "RemoveContainer" containerID="67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb"
Sep 17 18:56:02.731978 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:56:02.731957    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb\": container with ID starting with 67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb not found: ID does not exist" containerID="67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb"
Sep 17 18:56:02.732073 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.731986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb"} err="failed to get container status \"67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb\": rpc error: code = NotFound desc = could not find container \"67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb\": container with ID starting with 67955e5c24709aebccf4fe6a2d430ff65e941c2444d6c1d0d248e8733de260bb not found: ID does not exist"
Sep 17 18:56:02.732073 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.732005    2566 scope.go:117] "RemoveContainer" containerID="f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585"
Sep 17 18:56:02.732276 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:56:02.732259    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585\": container with ID starting with f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585 not found: ID does not exist" containerID="f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585"
Sep 17 18:56:02.732333 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.732282    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585"} err="failed to get container status \"f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585\": rpc error: code = NotFound desc = could not find container \"f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585\": container with ID starting with f8b3aec420eab685d8374da0de5aa356afd3b005ce331a8c2f2dd0732f5b6585 not found: ID does not exist"
Sep 17 18:56:02.732333 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.732297    2566 scope.go:117] "RemoveContainer" containerID="15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02"
Sep 17 18:56:02.732587 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:56:02.732566    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02\": container with ID starting with 15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02 not found: ID does not exist" containerID="15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02"
Sep 17 18:56:02.732635 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.732596    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02"} err="failed to get container status \"15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02\": rpc error: code = NotFound desc = could not find container \"15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02\": container with ID starting with 15d43e81667974e1d90912e35cc6f0c6545a62bbbc3b56115d820f5c8ed7ac02 not found: ID does not exist"
Sep 17 18:56:02.740165 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.740130    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"]
Sep 17 18:56:02.744128 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.744099    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bae88-predictor-c45b49bd9-slq57"]
Sep 17 18:56:02.755286 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.755268    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:56:02.780185 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.780074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-kube-rbac-proxy-sar-config\") pod \"40d24367-4972-4b7b-b6ac-8340d427c873\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") "
Sep 17 18:56:02.780185 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.780120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d24367-4972-4b7b-b6ac-8340d427c873-kserve-provision-location\") pod \"40d24367-4972-4b7b-b6ac-8340d427c873\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") "
Sep 17 18:56:02.780185 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.780166    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls\") pod \"40d24367-4972-4b7b-b6ac-8340d427c873\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") "
Sep 17 18:56:02.780452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.780243    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-cabundle-cert\") pod \"40d24367-4972-4b7b-b6ac-8340d427c873\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") "
Sep 17 18:56:02.780452 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.780281    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2627\" (UniqueName: \"kubernetes.io/projected/40d24367-4972-4b7b-b6ac-8340d427c873-kube-api-access-z2627\") pod \"40d24367-4972-4b7b-b6ac-8340d427c873\" (UID: \"40d24367-4972-4b7b-b6ac-8340d427c873\") "
Sep 17 18:56:02.780986 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.780954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d24367-4972-4b7b-b6ac-8340d427c873-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40d24367-4972-4b7b-b6ac-8340d427c873" (UID: "40d24367-4972-4b7b-b6ac-8340d427c873"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:02.781099 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.781018    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40d24367-4972-4b7b-b6ac-8340d427c873" (UID: "40d24367-4972-4b7b-b6ac-8340d427c873"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:02.781099 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.781070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40d24367-4972-4b7b-b6ac-8340d427c873" (UID: "40d24367-4972-4b7b-b6ac-8340d427c873"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:02.783034 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.782989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d24367-4972-4b7b-b6ac-8340d427c873-kube-api-access-z2627" (OuterVolumeSpecName: "kube-api-access-z2627") pod "40d24367-4972-4b7b-b6ac-8340d427c873" (UID: "40d24367-4972-4b7b-b6ac-8340d427c873"). InnerVolumeSpecName "kube-api-access-z2627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:02.783380 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.783356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40d24367-4972-4b7b-b6ac-8340d427c873" (UID: "40d24367-4972-4b7b-b6ac-8340d427c873"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:02.881059 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.881038    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.881059 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.881060    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2627\" (UniqueName: \"kubernetes.io/projected/40d24367-4972-4b7b-b6ac-8340d427c873-kube-api-access-z2627\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.881198 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.881071    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d24367-4972-4b7b-b6ac-8340d427c873-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.881198 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.881080    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d24367-4972-4b7b-b6ac-8340d427c873-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.881198 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:02.881089    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d24367-4972-4b7b-b6ac-8340d427c873-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:56:03.198258 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.198190    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" path="/var/lib/kubelet/pods/d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31/volumes"
Sep 17 18:56:03.710301 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.710272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"
Sep 17 18:56:03.710780 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.710299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927" event={"ID":"40d24367-4972-4b7b-b6ac-8340d427c873","Type":"ContainerDied","Data":"5d82c4ac36434fcf3bf9aaa5f44eb48fe8e0eee20948e310896ad1c6d70b9d86"}
Sep 17 18:56:03.710780 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.710377    2566 scope.go:117] "RemoveContainer" containerID="861f50c1ad7cce1651ed6d9afde5c092a2e88ec4c141147f893dc710aae51893"
Sep 17 18:56:03.713245 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.713223    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerID="b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30" exitCode=0
Sep 17 18:56:03.713343 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.713258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerDied","Data":"b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30"}
Sep 17 18:56:03.719284 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.719264    2566 scope.go:117] "RemoveContainer" containerID="273f4ede020f8e93be0de1d452149febab0a6f4c64071bb26185cf74a97c9572"
Sep 17 18:56:03.728632 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.728615    2566 scope.go:117] "RemoveContainer" containerID="1d95bc161fefe27ebc75fada1812231b1601ad03663fa23427d1deaa31f7ca92"
Sep 17 18:56:03.737809 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.737787    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"]
Sep 17 18:56:03.739967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:03.739946    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bae88-predictor-6d686494b5-b6927"]
Sep 17 18:56:04.719691 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:04.719655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerStarted","Data":"b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602"}
Sep 17 18:56:04.720152 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:04.719716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerStarted","Data":"903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40"}
Sep 17 18:56:04.720152 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:04.719921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:56:04.749705 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:04.749653    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podStartSLOduration=2.929420023 podStartE2EDuration="6.749638778s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820218755s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:04.748109508 +0000 UTC m=+1041.600813764" watchObservedRunningTime="2026-09-17 18:56:04.749638778 +0000 UTC m=+1041.602343022"
Sep 17 18:56:05.202703 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:05.202670    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" path="/var/lib/kubelet/pods/40d24367-4972-4b7b-b6ac-8340d427c873/volumes"
Sep 17 18:56:05.725575 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:05.725548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:56:05.726967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:05.726932    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:56:06.729781 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:06.729735    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:56:11.623669 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:11.623636    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:56:11.735124 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:11.735098    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:56:11.735597 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:11.735575    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:56:21.623301 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:21.623259    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:56:21.735846 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:21.735789    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:56:31.623846 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:31.623750    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:56:31.736057 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:31.736026    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:56:41.624126 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:41.624081    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:56:41.735969 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:41.735936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:56:51.624080 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:51.624035    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:56:51.736225 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:56:51.736187    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:57:01.623789 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:01.623759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:57:01.735985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:01.735955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:57:07.989782 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:07.989746    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"]
Sep 17 18:57:07.990323 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:07.989998    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" containerID="cri-o://c44d0a4dde51bf9eb429d32d1d379adeb53232b03ad36d019a9ca9bf864017eb" gracePeriod=30
Sep 17 18:57:07.990323 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:07.990048    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kube-rbac-proxy" containerID="cri-o://fe6e8411da32424e5a74448a66c18471127bd0f05c53ffad4bf37e7bc56d615b" gracePeriod=30
Sep 17 18:57:09.009354 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:09.009321    2566 generic.go:356] "Generic (PLEG): container finished" podID="30c0b269-160d-4534-b4a3-c51a5929e516" containerID="fe6e8411da32424e5a74448a66c18471127bd0f05c53ffad4bf37e7bc56d615b" exitCode=2
Sep 17 18:57:09.009714 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:09.009393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerDied","Data":"fe6e8411da32424e5a74448a66c18471127bd0f05c53ffad4bf37e7bc56d615b"}
Sep 17 18:57:11.618257 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:11.618211    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 18:57:11.623554 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:11.623524    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:57:11.735541 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:11.735503    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:57:12.023123 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.023088    2566 generic.go:356] "Generic (PLEG): container finished" podID="30c0b269-160d-4534-b4a3-c51a5929e516" containerID="c44d0a4dde51bf9eb429d32d1d379adeb53232b03ad36d019a9ca9bf864017eb" exitCode=0
Sep 17 18:57:12.023274 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.023156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerDied","Data":"c44d0a4dde51bf9eb429d32d1d379adeb53232b03ad36d019a9ca9bf864017eb"}
Sep 17 18:57:12.129300 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.129277    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:57:12.204755 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.204690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sw7m\" (UniqueName: \"kubernetes.io/projected/30c0b269-160d-4534-b4a3-c51a5929e516-kube-api-access-4sw7m\") pod \"30c0b269-160d-4534-b4a3-c51a5929e516\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") "
Sep 17 18:57:12.204755 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.204734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-cabundle-cert\") pod \"30c0b269-160d-4534-b4a3-c51a5929e516\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") "
Sep 17 18:57:12.204921 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.204766    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c0b269-160d-4534-b4a3-c51a5929e516-kserve-provision-location\") pod \"30c0b269-160d-4534-b4a3-c51a5929e516\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") "
Sep 17 18:57:12.204921 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.204879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-kube-rbac-proxy-sar-config\") pod \"30c0b269-160d-4534-b4a3-c51a5929e516\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") "
Sep 17 18:57:12.205031 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.204961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls\") pod \"30c0b269-160d-4534-b4a3-c51a5929e516\" (UID: \"30c0b269-160d-4534-b4a3-c51a5929e516\") "
Sep 17 18:57:12.205088 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.205043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c0b269-160d-4534-b4a3-c51a5929e516-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30c0b269-160d-4534-b4a3-c51a5929e516" (UID: "30c0b269-160d-4534-b4a3-c51a5929e516"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:12.205144 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.205111    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30c0b269-160d-4534-b4a3-c51a5929e516" (UID: "30c0b269-160d-4534-b4a3-c51a5929e516"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:12.205236 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.205220    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.205336 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.205239    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c0b269-160d-4534-b4a3-c51a5929e516-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.205336 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.205252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30c0b269-160d-4534-b4a3-c51a5929e516" (UID: "30c0b269-160d-4534-b4a3-c51a5929e516"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:12.207034 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.207004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c0b269-160d-4534-b4a3-c51a5929e516-kube-api-access-4sw7m" (OuterVolumeSpecName: "kube-api-access-4sw7m") pod "30c0b269-160d-4534-b4a3-c51a5929e516" (UID: "30c0b269-160d-4534-b4a3-c51a5929e516"). InnerVolumeSpecName "kube-api-access-4sw7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:12.207257 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.207227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30c0b269-160d-4534-b4a3-c51a5929e516" (UID: "30c0b269-160d-4534-b4a3-c51a5929e516"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:12.305753 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.305732    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c0b269-160d-4534-b4a3-c51a5929e516-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.305753 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.305751    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sw7m\" (UniqueName: \"kubernetes.io/projected/30c0b269-160d-4534-b4a3-c51a5929e516-kube-api-access-4sw7m\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:12.305905 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:12.305760    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c0b269-160d-4534-b4a3-c51a5929e516-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:13.029704 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.029674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf" event={"ID":"30c0b269-160d-4534-b4a3-c51a5929e516","Type":"ContainerDied","Data":"cfead21f3bc6b8be31475d14137582cab69ab55a925c1cf50bfceb2cfa7b1b81"}
Sep 17 18:57:13.030129 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.029714    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"
Sep 17 18:57:13.030129 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.029739    2566 scope.go:117] "RemoveContainer" containerID="fe6e8411da32424e5a74448a66c18471127bd0f05c53ffad4bf37e7bc56d615b"
Sep 17 18:57:13.039230 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.039214    2566 scope.go:117] "RemoveContainer" containerID="c44d0a4dde51bf9eb429d32d1d379adeb53232b03ad36d019a9ca9bf864017eb"
Sep 17 18:57:13.046514 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.046487    2566 scope.go:117] "RemoveContainer" containerID="7a2132fe6c8d0f77b650068ced801ca9139fcb20c718877fe8aaca00f9ef8393"
Sep 17 18:57:13.054471 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.054452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"]
Sep 17 18:57:13.058911 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.058891    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-4srhf"]
Sep 17 18:57:13.198487 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:13.198462    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" path="/var/lib/kubelet/pods/30c0b269-160d-4534-b4a3-c51a5929e516/volumes"
Sep 17 18:57:21.736527 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:21.736492    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:57:48.297238 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.297203    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"]
Sep 17 18:57:48.297680 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.297471    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" containerID="cri-o://903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40" gracePeriod=30
Sep 17 18:57:48.297680 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.297528    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kube-rbac-proxy" containerID="cri-o://b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602" gracePeriod=30
Sep 17 18:57:48.454321 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454292    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"]
Sep 17 18:57:48.454599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454586    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454600    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454610    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454615    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454622    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454627    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454634    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="storage-initializer"
Sep 17 18:57:48.454645 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454640    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="storage-initializer"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454649    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454654    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454664    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454669    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454678    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="storage-initializer"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454682    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="storage-initializer"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454693    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454697    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454703    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="storage-initializer"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454708    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="storage-initializer"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454755    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454763    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454770    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c0b269-160d-4534-b4a3-c51a5929e516" containerName="kserve-container"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454775    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kube-rbac-proxy"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454781    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73fbbd0-2aeb-4cd8-9b68-1b53c66c6d31" containerName="kserve-container"
Sep 17 18:57:48.454886 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.454786    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d24367-4972-4b7b-b6ac-8340d427c873" containerName="kserve-container"
Sep 17 18:57:48.457716 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.457702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.460530 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.460504    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-47a1c-kube-rbac-proxy-sar-config\""
Sep 17 18:57:48.460656 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.460503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-47a1c-predictor-serving-cert\""
Sep 17 18:57:48.469586 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.469564    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"]
Sep 17 18:57:48.575971 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.575946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fwd\" (UniqueName: \"kubernetes.io/projected/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-api-access-h2fwd\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.576102 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.575973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.576102 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.576026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11d540f-27cb-4f59-b044-3deb4f14de50-proxy-tls\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.677079 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.677044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fwd\" (UniqueName: \"kubernetes.io/projected/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-api-access-h2fwd\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.677079 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.677081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.677231 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.677130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11d540f-27cb-4f59-b044-3deb4f14de50-proxy-tls\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.677807 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.677789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.679543 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.679526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11d540f-27cb-4f59-b044-3deb4f14de50-proxy-tls\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.685998 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.685978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fwd\" (UniqueName: \"kubernetes.io/projected/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-api-access-h2fwd\") pod \"message-dumper-raw-47a1c-predictor-579b9d56d4-584vw\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") " pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:48.767651 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:48.767625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:49.038381 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:49.038359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"]
Sep 17 18:57:49.040282 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:57:49.040249    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11d540f_27cb_4f59_b044_3deb4f14de50.slice/crio-d5d0467679125ab1ca064c0315a5c8c3a1c4c409e22eda47bfc6b7ff898f7ed4 WatchSource:0}: Error finding container d5d0467679125ab1ca064c0315a5c8c3a1c4c409e22eda47bfc6b7ff898f7ed4: Status 404 returned error can't find the container with id d5d0467679125ab1ca064c0315a5c8c3a1c4c409e22eda47bfc6b7ff898f7ed4
Sep 17 18:57:49.042278 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:49.042257    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:57:49.184650 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:49.184622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" event={"ID":"b11d540f-27cb-4f59-b044-3deb4f14de50","Type":"ContainerStarted","Data":"d5d0467679125ab1ca064c0315a5c8c3a1c4c409e22eda47bfc6b7ff898f7ed4"}
Sep 17 18:57:49.186686 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:49.186661    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerID="b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602" exitCode=2
Sep 17 18:57:49.186789 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:49.186728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerDied","Data":"b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602"}
Sep 17 18:57:50.192355 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:50.192324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" event={"ID":"b11d540f-27cb-4f59-b044-3deb4f14de50","Type":"ContainerStarted","Data":"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"}
Sep 17 18:57:51.198421 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.198389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" event={"ID":"b11d540f-27cb-4f59-b044-3deb4f14de50","Type":"ContainerStarted","Data":"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"}
Sep 17 18:57:51.198865 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.198443    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:51.198865 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.198592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:51.200361 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.200344    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:57:51.224211 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.224171    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" podStartSLOduration=2.140748235 podStartE2EDuration="3.224158884s" totalImagesPullingTime="1.083410649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:51.223322493 +0000 UTC m=+1148.076026748" watchObservedRunningTime="2026-09-17 18:57:51.224158884 +0000 UTC m=+1148.076863129"
Sep 17 18:57:51.730599 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.730562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 18:57:51.736021 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:51.735989    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:57:52.534949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.534927    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:57:52.610017 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.609952    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-cabundle-cert\") pod \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") "
Sep 17 18:57:52.610017 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.609985    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d0b861-c13d-475f-a8b1-dc8840fa98df-proxy-tls\") pod \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") "
Sep 17 18:57:52.610195 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kserve-provision-location\") pod \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") "
Sep 17 18:57:52.610195 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wdtb\" (UniqueName: \"kubernetes.io/projected/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-api-access-7wdtb\") pod \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") "
Sep 17 18:57:52.610195 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-rbac-proxy-sar-config\") pod \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\" (UID: \"d5d0b861-c13d-475f-a8b1-dc8840fa98df\") "
Sep 17 18:57:52.610336 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5d0b861-c13d-475f-a8b1-dc8840fa98df" (UID: "d5d0b861-c13d-475f-a8b1-dc8840fa98df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:52.610399 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5d0b861-c13d-475f-a8b1-dc8840fa98df" (UID: "d5d0b861-c13d-475f-a8b1-dc8840fa98df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:52.610476 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610425    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.610541 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.610499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5d0b861-c13d-475f-a8b1-dc8840fa98df" (UID: "d5d0b861-c13d-475f-a8b1-dc8840fa98df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:52.612035 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.612016    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d0b861-c13d-475f-a8b1-dc8840fa98df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5d0b861-c13d-475f-a8b1-dc8840fa98df" (UID: "d5d0b861-c13d-475f-a8b1-dc8840fa98df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:52.612123 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.612096    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-api-access-7wdtb" (OuterVolumeSpecName: "kube-api-access-7wdtb") pod "d5d0b861-c13d-475f-a8b1-dc8840fa98df" (UID: "d5d0b861-c13d-475f-a8b1-dc8840fa98df"). InnerVolumeSpecName "kube-api-access-7wdtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:52.711551 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.711519    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.711551 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.711550    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d0b861-c13d-475f-a8b1-dc8840fa98df-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.711551 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.711560    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d0b861-c13d-475f-a8b1-dc8840fa98df-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:52.711712 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:52.711569    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdtb\" (UniqueName: \"kubernetes.io/projected/d5d0b861-c13d-475f-a8b1-dc8840fa98df-kube-api-access-7wdtb\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:57:53.208781 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.208744    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerID="903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40" exitCode=0
Sep 17 18:57:53.208949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.208816    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"
Sep 17 18:57:53.208949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.208853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerDied","Data":"903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40"}
Sep 17 18:57:53.208949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.208920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l" event={"ID":"d5d0b861-c13d-475f-a8b1-dc8840fa98df","Type":"ContainerDied","Data":"a40ecf989fc8be9e90f774a5b97d99fbbc91f10d7238d99251c8f19660f08e8a"}
Sep 17 18:57:53.208949 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.208944    2566 scope.go:117] "RemoveContainer" containerID="b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602"
Sep 17 18:57:53.222031 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.222006    2566 scope.go:117] "RemoveContainer" containerID="903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40"
Sep 17 18:57:53.230639 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.230614    2566 scope.go:117] "RemoveContainer" containerID="b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30"
Sep 17 18:57:53.231352 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.231328    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"]
Sep 17 18:57:53.235317 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.235296    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-8b7b7-predictor-6bc97c6c89-4hr5l"]
Sep 17 18:57:53.238511 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.238480    2566 scope.go:117] "RemoveContainer" containerID="b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602"
Sep 17 18:57:53.238975 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:57:53.238951    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602\": container with ID starting with b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602 not found: ID does not exist" containerID="b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602"
Sep 17 18:57:53.239072 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.239003    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602"} err="failed to get container status \"b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602\": rpc error: code = NotFound desc = could not find container \"b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602\": container with ID starting with b8197ae0b0685e8d6a1b113ceabe458fb44b67645e12d298ceb44005f0ddf602 not found: ID does not exist"
Sep 17 18:57:53.239072 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.239045    2566 scope.go:117] "RemoveContainer" containerID="903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40"
Sep 17 18:57:53.239410 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:57:53.239385    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40\": container with ID starting with 903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40 not found: ID does not exist" containerID="903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40"
Sep 17 18:57:53.239482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.239431    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40"} err="failed to get container status \"903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40\": rpc error: code = NotFound desc = could not find container \"903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40\": container with ID starting with 903c5f8c56f93fd63e3c4d40a6fa449ac4cfbee434f5757f4750b631791b8c40 not found: ID does not exist"
Sep 17 18:57:53.239482 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.239470    2566 scope.go:117] "RemoveContainer" containerID="b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30"
Sep 17 18:57:53.239802 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:57:53.239782    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30\": container with ID starting with b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30 not found: ID does not exist" containerID="b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30"
Sep 17 18:57:53.239864 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:53.239809    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30"} err="failed to get container status \"b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30\": rpc error: code = NotFound desc = could not find container \"b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30\": container with ID starting with b5a622d3eb51d3000a7ecf0ec51e3cbe4809e647e806bc866f45f463363c6b30 not found: ID does not exist"
Sep 17 18:57:55.199026 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:55.198990    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" path="/var/lib/kubelet/pods/d5d0b861-c13d-475f-a8b1-dc8840fa98df/volumes"
Sep 17 18:57:58.213821 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:57:58.213732    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:58:45.209095 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:58:45.209065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:58:45.209722 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:58:45.209705    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 18:59:03.376444 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376415    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"]
Sep 17 18:59:03.376798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376738    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kube-rbac-proxy"
Sep 17 18:59:03.376798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376750    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kube-rbac-proxy"
Sep 17 18:59:03.376798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376762    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="storage-initializer"
Sep 17 18:59:03.376798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376768    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="storage-initializer"
Sep 17 18:59:03.376798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376774    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container"
Sep 17 18:59:03.376798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376779    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container"
Sep 17 18:59:03.377012 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376853    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kserve-container"
Sep 17 18:59:03.377012 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.376860    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d0b861-c13d-475f-a8b1-dc8840fa98df" containerName="kube-rbac-proxy"
Sep 17 18:59:03.380021 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.380001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.382402 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.382375    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-bfe648-predictor-serving-cert\""
Sep 17 18:59:03.382402 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.382375    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-bfe648\""
Sep 17 18:59:03.382589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.382382    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-bfe648-dockercfg-z8md5\""
Sep 17 18:59:03.382589 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.382419    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:59:03.382707 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.382694    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-bfe648-kube-rbac-proxy-sar-config\""
Sep 17 18:59:03.393573 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.393553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"]
Sep 17 18:59:03.409644 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.409624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.409754 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.409662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l2rs\" (UniqueName: \"kubernetes.io/projected/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-api-access-2l2rs\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.409754 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.409694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5613d9-0bf2-4d2f-a002-9ab41598a814-proxy-tls\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.409856 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.409774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-cabundle-cert\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.409856 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.409822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kserve-provision-location\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510259 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-cabundle-cert\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510359 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kserve-provision-location\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510359 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510359 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l2rs\" (UniqueName: \"kubernetes.io/projected/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-api-access-2l2rs\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510502 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5613d9-0bf2-4d2f-a002-9ab41598a814-proxy-tls\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510758 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kserve-provision-location\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.510944 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-cabundle-cert\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.511024 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.510949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.512677 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.512659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5613d9-0bf2-4d2f-a002-9ab41598a814-proxy-tls\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.518798 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.518778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l2rs\" (UniqueName: \"kubernetes.io/projected/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-api-access-2l2rs\") pod \"isvc-secondary-bfe648-predictor-554f8ff779-4c7jr\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") " pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.689729 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.689671    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:03.970689 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:03.970629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"]
Sep 17 18:59:03.972788 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:59:03.972762    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5613d9_0bf2_4d2f_a002_9ab41598a814.slice/crio-32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3 WatchSource:0}: Error finding container 32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3: Status 404 returned error can't find the container with id 32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3
Sep 17 18:59:04.496865 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:04.496814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" event={"ID":"2f5613d9-0bf2-4d2f-a002-9ab41598a814","Type":"ContainerStarted","Data":"67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832"}
Sep 17 18:59:04.497225 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:04.496889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" event={"ID":"2f5613d9-0bf2-4d2f-a002-9ab41598a814","Type":"ContainerStarted","Data":"32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3"}
Sep 17 18:59:06.506068 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:06.506043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/0.log"
Sep 17 18:59:06.506429 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:06.506081    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerID="67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832" exitCode=1
Sep 17 18:59:06.506429 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:06.506126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" event={"ID":"2f5613d9-0bf2-4d2f-a002-9ab41598a814","Type":"ContainerDied","Data":"67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832"}
Sep 17 18:59:07.512402 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:07.512374    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/0.log"
Sep 17 18:59:07.512787 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:07.512495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" event={"ID":"2f5613d9-0bf2-4d2f-a002-9ab41598a814","Type":"ContainerStarted","Data":"c56cf6ea1e6d7266d253d0283cac4941abd3552e27f63921dd1d2c10db0e28a3"}
Sep 17 18:59:11.531667 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.531591    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/1.log"
Sep 17 18:59:11.532115 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.532098    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/0.log"
Sep 17 18:59:11.532168 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.532134    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerID="c56cf6ea1e6d7266d253d0283cac4941abd3552e27f63921dd1d2c10db0e28a3" exitCode=1
Sep 17 18:59:11.532221 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.532169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" event={"ID":"2f5613d9-0bf2-4d2f-a002-9ab41598a814","Type":"ContainerDied","Data":"c56cf6ea1e6d7266d253d0283cac4941abd3552e27f63921dd1d2c10db0e28a3"}
Sep 17 18:59:11.532264 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.532227    2566 scope.go:117] "RemoveContainer" containerID="67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832"
Sep 17 18:59:11.532631 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.532598    2566 scope.go:117] "RemoveContainer" containerID="67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832"
Sep 17 18:59:11.543577 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:59:11.543553    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_kserve-ci-e2e-test(2f5613d9-0bf2-4d2f-a002-9ab41598a814)\"" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814"
Sep 17 18:59:11.543779 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:59:11.543751    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_kserve-ci-e2e-test_2f5613d9-0bf2-4d2f-a002-9ab41598a814_0 in pod sandbox 32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3 from index: no such id: '67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832'" containerID="67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832"
Sep 17 18:59:11.543861 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:11.543791    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_kserve-ci-e2e-test_2f5613d9-0bf2-4d2f-a002-9ab41598a814_0 in pod sandbox 32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3 from index: no such id: '67116c1908227e6b92df2bdf584052d9d82e2165aa5bb55437c8f12ae904f832'"
Sep 17 18:59:12.537690 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:12.537662    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/1.log"
Sep 17 18:59:17.412695 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.412663    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"]
Sep 17 18:59:17.546408 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.546387    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/1.log"
Sep 17 18:59:17.550009 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.549994    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:17.557466 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.557447    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bfe648-predictor-554f8ff779-4c7jr_2f5613d9-0bf2-4d2f-a002-9ab41598a814/storage-initializer/1.log"
Sep 17 18:59:17.557569 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.557502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr" event={"ID":"2f5613d9-0bf2-4d2f-a002-9ab41598a814","Type":"ContainerDied","Data":"32c390e5f9319797cd727e85a633dbc8eb622c6a06cd5facfdccc94c42e4c4a3"}
Sep 17 18:59:17.557569 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.557521    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"
Sep 17 18:59:17.557569 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.557549    2566 scope.go:117] "RemoveContainer" containerID="c56cf6ea1e6d7266d253d0283cac4941abd3552e27f63921dd1d2c10db0e28a3"
Sep 17 18:59:17.606839 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.606804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-cabundle-cert\") pod \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") "
Sep 17 18:59:17.606985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.606847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kserve-provision-location\") pod \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") "
Sep 17 18:59:17.606985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.606878    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-rbac-proxy-sar-config\") pod \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") "
Sep 17 18:59:17.606985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.606939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5613d9-0bf2-4d2f-a002-9ab41598a814-proxy-tls\") pod \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") "
Sep 17 18:59:17.606985 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.606957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l2rs\" (UniqueName: \"kubernetes.io/projected/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-api-access-2l2rs\") pod \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\" (UID: \"2f5613d9-0bf2-4d2f-a002-9ab41598a814\") "
Sep 17 18:59:17.607208 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.607160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f5613d9-0bf2-4d2f-a002-9ab41598a814" (UID: "2f5613d9-0bf2-4d2f-a002-9ab41598a814"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:17.607208 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.607181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f5613d9-0bf2-4d2f-a002-9ab41598a814" (UID: "2f5613d9-0bf2-4d2f-a002-9ab41598a814"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:17.607291 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.607226    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f5613d9-0bf2-4d2f-a002-9ab41598a814" (UID: "2f5613d9-0bf2-4d2f-a002-9ab41598a814"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:17.608905 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.608881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5613d9-0bf2-4d2f-a002-9ab41598a814-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f5613d9-0bf2-4d2f-a002-9ab41598a814" (UID: "2f5613d9-0bf2-4d2f-a002-9ab41598a814"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:17.608991 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.608958    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-api-access-2l2rs" (OuterVolumeSpecName: "kube-api-access-2l2rs") pod "2f5613d9-0bf2-4d2f-a002-9ab41598a814" (UID: "2f5613d9-0bf2-4d2f-a002-9ab41598a814"). InnerVolumeSpecName "kube-api-access-2l2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:17.707389 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.707334    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:17.707389 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.707358    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:17.707389 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.707372    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:17.707389 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.707386    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5613d9-0bf2-4d2f-a002-9ab41598a814-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:17.707570 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.707401    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l2rs\" (UniqueName: \"kubernetes.io/projected/2f5613d9-0bf2-4d2f-a002-9ab41598a814-kube-api-access-2l2rs\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:17.895218 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.895193    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"]
Sep 17 18:59:17.901510 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:17.901487    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bfe648-predictor-554f8ff779-4c7jr"]
Sep 17 18:59:19.198950 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:19.198909    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" path="/var/lib/kubelet/pods/2f5613d9-0bf2-4d2f-a002-9ab41598a814/volumes"
Sep 17 18:59:43.498309 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:43.498203    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-47a1c-predictor-579b9d56d4-584vw_b11d540f-27cb-4f59-b044-3deb4f14de50/kserve-container/0.log"
Sep 17 18:59:43.825967 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:43.825937    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"]
Sep 17 18:59:43.826165 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:43.826140    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kserve-container" containerID="cri-o://78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f" gracePeriod=30
Sep 17 18:59:43.826236 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:43.826200    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kube-rbac-proxy" containerID="cri-o://4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413" gracePeriod=30
Sep 17 18:59:44.013480 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.013453    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"]
Sep 17 18:59:44.013899 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.013879    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerName="storage-initializer"
Sep 17 18:59:44.013899 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.013900    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerName="storage-initializer"
Sep 17 18:59:44.014070 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.013920    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerName="storage-initializer"
Sep 17 18:59:44.014070 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.013927    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerName="storage-initializer"
Sep 17 18:59:44.014070 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.014022    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerName="storage-initializer"
Sep 17 18:59:44.014070 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.014039    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5613d9-0bf2-4d2f-a002-9ab41598a814" containerName="storage-initializer"
Sep 17 18:59:44.017373 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.017351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.020419 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.020400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1b147-predictor-serving-cert\""
Sep 17 18:59:44.020844 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.020812    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:59:44.020975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.020939    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1b147-kube-rbac-proxy-sar-config\""
Sep 17 18:59:44.020975 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.020951    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:59:44.032311 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.032284    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"]
Sep 17 18:59:44.071487 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.071467    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:59:44.095940 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.095890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.096025 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.095938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.096070 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.096038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.096070 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.096064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-api-access-tc9m9\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.096134 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.096088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0967bfa8-36ad-48f7-af94-5713a99cc793-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.197242 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197218    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2fwd\" (UniqueName: \"kubernetes.io/projected/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-api-access-h2fwd\") pod \"b11d540f-27cb-4f59-b044-3deb4f14de50\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") "
Sep 17 18:59:44.197342 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197266    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11d540f-27cb-4f59-b044-3deb4f14de50-proxy-tls\") pod \"b11d540f-27cb-4f59-b044-3deb4f14de50\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") "
Sep 17 18:59:44.197342 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-rbac-proxy-sar-config\") pod \"b11d540f-27cb-4f59-b044-3deb4f14de50\" (UID: \"b11d540f-27cb-4f59-b044-3deb4f14de50\") "
Sep 17 18:59:44.197428 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.197468 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-api-access-tc9m9\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.197522 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0967bfa8-36ad-48f7-af94-5713a99cc793-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.197522 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.197622 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.197716 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:59:44.197699    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-serving-cert: secret "isvc-sklearn-scale-raw-1b147-predictor-serving-cert" not found
Sep 17 18:59:44.197781 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b11d540f-27cb-4f59-b044-3deb4f14de50" (UID: "b11d540f-27cb-4f59-b044-3deb4f14de50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:44.197781 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:59:44.197763    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls podName:0967bfa8-36ad-48f7-af94-5713a99cc793 nodeName:}" failed. No retries permitted until 2026-09-17 18:59:44.697744298 +0000 UTC m=+1261.550448533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls") pod "isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" (UID: "0967bfa8-36ad-48f7-af94-5713a99cc793") : secret "isvc-sklearn-scale-raw-1b147-predictor-serving-cert" not found
Sep 17 18:59:44.197984 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.197957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0967bfa8-36ad-48f7-af94-5713a99cc793-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.198358 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.198336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.198438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.198338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.199872 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.199847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11d540f-27cb-4f59-b044-3deb4f14de50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b11d540f-27cb-4f59-b044-3deb4f14de50" (UID: "b11d540f-27cb-4f59-b044-3deb4f14de50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:44.199872 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.199851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-api-access-h2fwd" (OuterVolumeSpecName: "kube-api-access-h2fwd") pod "b11d540f-27cb-4f59-b044-3deb4f14de50" (UID: "b11d540f-27cb-4f59-b044-3deb4f14de50"). InnerVolumeSpecName "kube-api-access-h2fwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:44.208394 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.208374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-api-access-tc9m9\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.298515 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.298490    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2fwd\" (UniqueName: \"kubernetes.io/projected/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-api-access-h2fwd\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:44.298603 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.298510    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11d540f-27cb-4f59-b044-3deb4f14de50-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:44.298603 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.298529    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11d540f-27cb-4f59-b044-3deb4f14de50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 18:59:44.668958 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.668926    2566 generic.go:356] "Generic (PLEG): container finished" podID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerID="4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413" exitCode=2
Sep 17 18:59:44.668958 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.668956    2566 generic.go:356] "Generic (PLEG): container finished" podID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerID="78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f" exitCode=2
Sep 17 18:59:44.669438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.668969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" event={"ID":"b11d540f-27cb-4f59-b044-3deb4f14de50","Type":"ContainerDied","Data":"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"}
Sep 17 18:59:44.669438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.669037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" event={"ID":"b11d540f-27cb-4f59-b044-3deb4f14de50","Type":"ContainerDied","Data":"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"}
Sep 17 18:59:44.669438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.669059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw" event={"ID":"b11d540f-27cb-4f59-b044-3deb4f14de50","Type":"ContainerDied","Data":"d5d0467679125ab1ca064c0315a5c8c3a1c4c409e22eda47bfc6b7ff898f7ed4"}
Sep 17 18:59:44.669438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.668982    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"
Sep 17 18:59:44.669438 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.669085    2566 scope.go:117] "RemoveContainer" containerID="4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"
Sep 17 18:59:44.677689 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.677670    2566 scope.go:117] "RemoveContainer" containerID="78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"
Sep 17 18:59:44.685484 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.685467    2566 scope.go:117] "RemoveContainer" containerID="4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"
Sep 17 18:59:44.685747 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:59:44.685729    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413\": container with ID starting with 4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413 not found: ID does not exist" containerID="4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"
Sep 17 18:59:44.685788 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.685758    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"} err="failed to get container status \"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413\": rpc error: code = NotFound desc = could not find container \"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413\": container with ID starting with 4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413 not found: ID does not exist"
Sep 17 18:59:44.685788 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.685774    2566 scope.go:117] "RemoveContainer" containerID="78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"
Sep 17 18:59:44.686007 ip-10-0-136-86 kubenswrapper[2566]: E0917 18:59:44.685991    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f\": container with ID starting with 78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f not found: ID does not exist" containerID="78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"
Sep 17 18:59:44.686047 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.686011    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"} err="failed to get container status \"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f\": rpc error: code = NotFound desc = could not find container \"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f\": container with ID starting with 78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f not found: ID does not exist"
Sep 17 18:59:44.686047 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.686024    2566 scope.go:117] "RemoveContainer" containerID="4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"
Sep 17 18:59:44.686220 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.686205    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413"} err="failed to get container status \"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413\": rpc error: code = NotFound desc = could not find container \"4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413\": container with ID starting with 4235686e53cd6f244f4bc660bd2713fdd2436adf7a50d5905e59b6d311f69413 not found: ID does not exist"
Sep 17 18:59:44.686256 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.686221    2566 scope.go:117] "RemoveContainer" containerID="78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"
Sep 17 18:59:44.686418 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.686400    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f"} err="failed to get container status \"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f\": rpc error: code = NotFound desc = could not find container \"78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f\": container with ID starting with 78742b7677f10bcfafdfd545eba7089d7cdfd8100ba4658d09d754c5f6aaf00f not found: ID does not exist"
Sep 17 18:59:44.692752 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.692731    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"]
Sep 17 18:59:44.696650 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.696632    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47a1c-predictor-579b9d56d4-584vw"]
Sep 17 18:59:44.701632 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.701616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.703838 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.703806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls\") pod \"isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:44.927990 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:44.927935    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:45.199213 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:45.199145    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" path="/var/lib/kubelet/pods/b11d540f-27cb-4f59-b044-3deb4f14de50/volumes"
Sep 17 18:59:45.203631 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:45.203612    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"]
Sep 17 18:59:45.204416 ip-10-0-136-86 kubenswrapper[2566]: W0917 18:59:45.204393    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0967bfa8_36ad_48f7_af94_5713a99cc793.slice/crio-7af50647a7a5a5f987cb2bb5f2d2370e82d28265b128a4703e9ef59845b93ba5 WatchSource:0}: Error finding container 7af50647a7a5a5f987cb2bb5f2d2370e82d28265b128a4703e9ef59845b93ba5: Status 404 returned error can't find the container with id 7af50647a7a5a5f987cb2bb5f2d2370e82d28265b128a4703e9ef59845b93ba5
Sep 17 18:59:45.208875 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:45.208858    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:59:45.674793 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:45.674753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerStarted","Data":"ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0"}
Sep 17 18:59:45.675184 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:45.674894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerStarted","Data":"7af50647a7a5a5f987cb2bb5f2d2370e82d28265b128a4703e9ef59845b93ba5"}
Sep 17 18:59:49.693352 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:49.693320    2566 generic.go:356] "Generic (PLEG): container finished" podID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerID="ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0" exitCode=0
Sep 17 18:59:49.695642 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:49.693399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerDied","Data":"ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0"}
Sep 17 18:59:50.699463 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:50.699432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerStarted","Data":"6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d"}
Sep 17 18:59:50.699877 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:50.699491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerStarted","Data":"e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf"}
Sep 17 18:59:50.699877 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:50.699677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:51.704025 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:51.703991    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:51.705230 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:51.705197    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:59:52.708157 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:52.708122    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:59:57.712460 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:57.712425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 18:59:57.713153 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:57.713121    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:59:57.736147 ip-10-0-136-86 kubenswrapper[2566]: I0917 18:59:57.736104    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podStartSLOduration=10.864318955 podStartE2EDuration="14.736090779s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.871771824s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:50.729768692 +0000 UTC m=+1267.582472946" watchObservedRunningTime="2026-09-17 18:59:57.736090779 +0000 UTC m=+1274.588795025"
Sep 17 19:00:07.713512 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:00:07.713466    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:00:17.713355 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:00:17.713308    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:00:27.712998 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:00:27.712955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:00:37.713455 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:00:37.713416    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:00:47.713275 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:00:47.713239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:00:57.713065 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:00:57.712981    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:01:07.714068 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:01:07.714026    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:01:17.713081 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:01:17.713038    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:01:25.195732 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:01:25.195690    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:01:35.195723 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:01:35.195676    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:01:45.195884 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:01:45.195847    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:01:55.196026 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:01:55.195979    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:02:05.195847 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:05.195787    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:02:15.199058 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:15.199026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 19:02:23.944976 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:23.944936    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"]
Sep 17 19:02:23.945412 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:23.945200    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" containerID="cri-o://e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf" gracePeriod=30
Sep 17 19:02:23.945412 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:23.945230    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kube-rbac-proxy" containerID="cri-o://6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d" gracePeriod=30
Sep 17 19:02:24.326721 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:24.326691    2566 generic.go:356] "Generic (PLEG): container finished" podID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerID="6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d" exitCode=2
Sep 17 19:02:24.326857 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:24.326749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerDied","Data":"6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d"}
Sep 17 19:02:25.196107 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:25.196068    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:02:27.708888 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:27.708781    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:02:29.200777 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.200745    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"]
Sep 17 19:02:29.201169 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.201137    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kube-rbac-proxy"
Sep 17 19:02:29.201169 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.201151    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kube-rbac-proxy"
Sep 17 19:02:29.201169 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.201168    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kserve-container"
Sep 17 19:02:29.201279 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.201174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kserve-container"
Sep 17 19:02:29.201279 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.201240    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kube-rbac-proxy"
Sep 17 19:02:29.201279 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.201257    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11d540f-27cb-4f59-b044-3deb4f14de50" containerName="kserve-container"
Sep 17 19:02:29.204334 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.204319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.206660 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.206639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 19:02:29.206810 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.206793    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 19:02:29.218924 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.218900    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"]
Sep 17 19:02:29.282625 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.282597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.282729 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.282634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.282729 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.282669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.282729 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.282703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.282895 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.282784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfp7\" (UniqueName: \"kubernetes.io/projected/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-api-access-vlfp7\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.383816 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.383792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.383965 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.383857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfp7\" (UniqueName: \"kubernetes.io/projected/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-api-access-vlfp7\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.383965 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.383881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.383965 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.383905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.384143 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:02:29.383991    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 19:02:29.384143 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.384022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.384143 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:02:29.384042    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls podName:d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98 nodeName:}" failed. No retries permitted until 2026-09-17 19:02:29.884024864 +0000 UTC m=+1426.736729091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls") pod "isvc-canary-create-predictor-b75cfc78d-9p9z8" (UID: "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 19:02:29.384334 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.384314    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.384542 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.384523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.384606 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.384542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.400861 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.400826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfp7\" (UniqueName: \"kubernetes.io/projected/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-api-access-vlfp7\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.887018 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.886990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:29.889334 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:29.889307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-9p9z8\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:30.114801 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:30.114774    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:30.389591 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:30.389566    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"]
Sep 17 19:02:30.392021 ip-10-0-136-86 kubenswrapper[2566]: W0917 19:02:30.391992    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77561b9_fc4e_41b5_bddd_0a2ba3ba8c98.slice/crio-e3cb5feac90b255efae7681a5b6347b9f66ee85049541fb8ec29e751557f2e99 WatchSource:0}: Error finding container e3cb5feac90b255efae7681a5b6347b9f66ee85049541fb8ec29e751557f2e99: Status 404 returned error can't find the container with id e3cb5feac90b255efae7681a5b6347b9f66ee85049541fb8ec29e751557f2e99
Sep 17 19:02:31.354161 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:31.354121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerStarted","Data":"a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb"}
Sep 17 19:02:31.354323 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:31.354181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerStarted","Data":"e3cb5feac90b255efae7681a5b6347b9f66ee85049541fb8ec29e751557f2e99"}
Sep 17 19:02:32.708612 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.708579    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:02:32.790876 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.790853    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 19:02:32.806739 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.806718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-cabundle-cert\") pod \"0967bfa8-36ad-48f7-af94-5713a99cc793\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") "
Sep 17 19:02:32.806817 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.806766    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-api-access-tc9m9\") pod \"0967bfa8-36ad-48f7-af94-5713a99cc793\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") "
Sep 17 19:02:32.806817 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.806801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-rbac-proxy-sar-config\") pod \"0967bfa8-36ad-48f7-af94-5713a99cc793\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") "
Sep 17 19:02:32.806949 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.806823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0967bfa8-36ad-48f7-af94-5713a99cc793-kserve-provision-location\") pod \"0967bfa8-36ad-48f7-af94-5713a99cc793\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") "
Sep 17 19:02:32.806949 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.806886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls\") pod \"0967bfa8-36ad-48f7-af94-5713a99cc793\" (UID: \"0967bfa8-36ad-48f7-af94-5713a99cc793\") "
Sep 17 19:02:32.807142 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.807112    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0967bfa8-36ad-48f7-af94-5713a99cc793" (UID: "0967bfa8-36ad-48f7-af94-5713a99cc793"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:32.807251 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.807140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0967bfa8-36ad-48f7-af94-5713a99cc793-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0967bfa8-36ad-48f7-af94-5713a99cc793" (UID: "0967bfa8-36ad-48f7-af94-5713a99cc793"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:32.807251 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.807187    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0967bfa8-36ad-48f7-af94-5713a99cc793" (UID: "0967bfa8-36ad-48f7-af94-5713a99cc793"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:32.808798 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.808778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-api-access-tc9m9" (OuterVolumeSpecName: "kube-api-access-tc9m9") pod "0967bfa8-36ad-48f7-af94-5713a99cc793" (UID: "0967bfa8-36ad-48f7-af94-5713a99cc793"). InnerVolumeSpecName "kube-api-access-tc9m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:32.808872 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.808811    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0967bfa8-36ad-48f7-af94-5713a99cc793" (UID: "0967bfa8-36ad-48f7-af94-5713a99cc793"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:32.908469 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.908410    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0967bfa8-36ad-48f7-af94-5713a99cc793-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:02:32.908469 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.908433    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:02:32.908469 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.908441    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-api-access-tc9m9\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:02:32.908469 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.908450    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0967bfa8-36ad-48f7-af94-5713a99cc793-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:02:32.908469 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:32.908459    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0967bfa8-36ad-48f7-af94-5713a99cc793-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:02:33.364158 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.364119    2566 generic.go:356] "Generic (PLEG): container finished" podID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerID="e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf" exitCode=0
Sep 17 19:02:33.364287 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.364178    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"
Sep 17 19:02:33.364287 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.364187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerDied","Data":"e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf"}
Sep 17 19:02:33.364287 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.364245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc" event={"ID":"0967bfa8-36ad-48f7-af94-5713a99cc793","Type":"ContainerDied","Data":"7af50647a7a5a5f987cb2bb5f2d2370e82d28265b128a4703e9ef59845b93ba5"}
Sep 17 19:02:33.364287 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.364271    2566 scope.go:117] "RemoveContainer" containerID="6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d"
Sep 17 19:02:33.372867 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.372851    2566 scope.go:117] "RemoveContainer" containerID="e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf"
Sep 17 19:02:33.379951 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.379920    2566 scope.go:117] "RemoveContainer" containerID="ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0"
Sep 17 19:02:33.383928 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.383905    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"]
Sep 17 19:02:33.387709 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.387688    2566 scope.go:117] "RemoveContainer" containerID="6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d"
Sep 17 19:02:33.387990 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:02:33.387969    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d\": container with ID starting with 6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d not found: ID does not exist" containerID="6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d"
Sep 17 19:02:33.388067 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.387997    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d"} err="failed to get container status \"6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d\": rpc error: code = NotFound desc = could not find container \"6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d\": container with ID starting with 6c3c5844ea9eff349de0a6a20ac9614974cf7a482cbeed5a43a296d757de849d not found: ID does not exist"
Sep 17 19:02:33.388067 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.388016    2566 scope.go:117] "RemoveContainer" containerID="e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf"
Sep 17 19:02:33.388328 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:02:33.388307    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf\": container with ID starting with e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf not found: ID does not exist" containerID="e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf"
Sep 17 19:02:33.388368 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.388334    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf"} err="failed to get container status \"e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf\": rpc error: code = NotFound desc = could not find container \"e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf\": container with ID starting with e5d5d3bb7c0ff6603fb20ea56bcb0ef6f13bdcce74a29cf7a82eb5f29d1c67cf not found: ID does not exist"
Sep 17 19:02:33.388368 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.388348    2566 scope.go:117] "RemoveContainer" containerID="ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0"
Sep 17 19:02:33.388569 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:02:33.388555    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0\": container with ID starting with ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0 not found: ID does not exist" containerID="ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0"
Sep 17 19:02:33.388612 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.388573    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0"} err="failed to get container status \"ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0\": rpc error: code = NotFound desc = could not find container \"ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0\": container with ID starting with ec757519b300c57e08143d18e51e7163eb9aefcc09d2498126bd51a21c1ed0c0 not found: ID does not exist"
Sep 17 19:02:33.389094 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:33.389079    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1b147-predictor-54fd889595-dsbxc"]
Sep 17 19:02:34.370385 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:34.370355    2566 generic.go:356] "Generic (PLEG): container finished" podID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerID="a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb" exitCode=0
Sep 17 19:02:34.370819 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:34.370439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerDied","Data":"a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb"}
Sep 17 19:02:35.198746 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.198712    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" path="/var/lib/kubelet/pods/0967bfa8-36ad-48f7-af94-5713a99cc793/volumes"
Sep 17 19:02:35.376607 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.376577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerStarted","Data":"3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9"}
Sep 17 19:02:35.376973 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.376636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerStarted","Data":"70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd"}
Sep 17 19:02:35.376973 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.376856    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:35.376973 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.376904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:35.378341 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.378316    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:02:35.401759 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:35.401718    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podStartSLOduration=2.708916721 podStartE2EDuration="6.40170235s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.692785629s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:35.401569236 +0000 UTC m=+1432.254273497" watchObservedRunningTime="2026-09-17 19:02:35.40170235 +0000 UTC m=+1432.254406594"
Sep 17 19:02:36.380624 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:36.380586    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:02:41.384890 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:41.384862    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:02:41.385355 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:41.385331    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:02:51.385749 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:02:51.385712    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:03:01.386301 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:01.386262    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:03:11.385763 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:11.385723    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:03:21.385804 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:21.385769    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:03:31.385508 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:31.385470    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:03:41.386045 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:41.386003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:03:45.235541 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:45.235511    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 19:03:45.237436 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:45.237410    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 19:03:51.385985 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:51.385957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:03:59.256453 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:59.256378    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"]
Sep 17 19:03:59.256915 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:59.256590    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" containerID="cri-o://70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd" gracePeriod=30
Sep 17 19:03:59.256915 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:59.256622    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kube-rbac-proxy" containerID="cri-o://3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9" gracePeriod=30
Sep 17 19:03:59.710971 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:59.710943    2566 generic.go:356] "Generic (PLEG): container finished" podID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerID="3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9" exitCode=2
Sep 17 19:03:59.711113 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:03:59.710978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerDied","Data":"3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9"}
Sep 17 19:04:01.381088 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:01.381047    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:04:01.385404 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:01.385376    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:04:03.301799 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.301776    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:04:03.354215 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlfp7\" (UniqueName: \"kubernetes.io/projected/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-api-access-vlfp7\") pod \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") "
Sep 17 19:04:03.354345 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-cabundle-cert\") pod \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") "
Sep 17 19:04:03.354345 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-rbac-proxy-sar-config\") pod \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") "
Sep 17 19:04:03.354345 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls\") pod \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") "
Sep 17 19:04:03.354523 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354371    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kserve-provision-location\") pod \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\" (UID: \"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98\") "
Sep 17 19:04:03.354788 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" (UID: "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:03.354907 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" (UID: "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:03.354907 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.354791    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" (UID: "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:03.356355 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.356333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" (UID: "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:03.356643 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.356623    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-api-access-vlfp7" (OuterVolumeSpecName: "kube-api-access-vlfp7") pod "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" (UID: "d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98"). InnerVolumeSpecName "kube-api-access-vlfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:03.455023 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.454968    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfp7\" (UniqueName: \"kubernetes.io/projected/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-api-access-vlfp7\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.455023 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.454990    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.455023 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.455000    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.455023 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.455009    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.455023 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.455019    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:04:03.730459 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.730375    2566 generic.go:356] "Generic (PLEG): container finished" podID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerID="70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd" exitCode=0
Sep 17 19:04:03.730459 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.730425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerDied","Data":"70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd"}
Sep 17 19:04:03.730459 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.730437    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"
Sep 17 19:04:03.730704 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.730507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8" event={"ID":"d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98","Type":"ContainerDied","Data":"e3cb5feac90b255efae7681a5b6347b9f66ee85049541fb8ec29e751557f2e99"}
Sep 17 19:04:03.730704 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.730555    2566 scope.go:117] "RemoveContainer" containerID="3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9"
Sep 17 19:04:03.739685 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.739669    2566 scope.go:117] "RemoveContainer" containerID="70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd"
Sep 17 19:04:03.746906 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.746890    2566 scope.go:117] "RemoveContainer" containerID="a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb"
Sep 17 19:04:03.753997 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.753981    2566 scope.go:117] "RemoveContainer" containerID="3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9"
Sep 17 19:04:03.754260 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:04:03.754241    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9\": container with ID starting with 3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9 not found: ID does not exist" containerID="3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9"
Sep 17 19:04:03.754302 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.754267    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9"} err="failed to get container status \"3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9\": rpc error: code = NotFound desc = could not find container \"3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9\": container with ID starting with 3427537ebff4afd0b9fa1cb1d8ba5cf4e3c87b96d4e9f4aafd444117fcb053e9 not found: ID does not exist"
Sep 17 19:04:03.754302 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.754283    2566 scope.go:117] "RemoveContainer" containerID="70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd"
Sep 17 19:04:03.754548 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:04:03.754532    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd\": container with ID starting with 70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd not found: ID does not exist" containerID="70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd"
Sep 17 19:04:03.754601 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.754549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd"} err="failed to get container status \"70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd\": rpc error: code = NotFound desc = could not find container \"70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd\": container with ID starting with 70271058e5286b439d5009ba5dc72f646cfb385427b87bb45c6c1b26ffdf71fd not found: ID does not exist"
Sep 17 19:04:03.754601 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.754563    2566 scope.go:117] "RemoveContainer" containerID="a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb"
Sep 17 19:04:03.754766 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:04:03.754748    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb\": container with ID starting with a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb not found: ID does not exist" containerID="a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb"
Sep 17 19:04:03.754811 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.754770    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb"} err="failed to get container status \"a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb\": rpc error: code = NotFound desc = could not find container \"a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb\": container with ID starting with a57639173d269cf42c1672e2ec58a36718ad68648e267b4dee15bbb3adab6edb not found: ID does not exist"
Sep 17 19:04:03.772636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.772616    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"]
Sep 17 19:04:03.792087 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:03.792066    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-9p9z8"]
Sep 17 19:04:04.813166 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813134    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"]
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813510    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813525    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813547    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813555    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813563    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="storage-initializer"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813569    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="storage-initializer"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813575    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kube-rbac-proxy"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813581    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kube-rbac-proxy"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813589    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kube-rbac-proxy"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813594    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kube-rbac-proxy"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813606    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="storage-initializer"
Sep 17 19:04:04.813636 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813611    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="storage-initializer"
Sep 17 19:04:04.814043 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813663    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kube-rbac-proxy"
Sep 17 19:04:04.814043 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813673    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kube-rbac-proxy"
Sep 17 19:04:04.814043 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813681    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" containerName="kserve-container"
Sep 17 19:04:04.814043 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.813687    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0967bfa8-36ad-48f7-af94-5713a99cc793" containerName="kserve-container"
Sep 17 19:04:04.817944 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.817926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.822215 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.822191    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:04:04.822314 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.822232    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 19:04:04.824456 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.824130    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 19:04:04.825138 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.825120    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:04:04.825226 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.825139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:04:04.825296 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.825275    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:04:04.825612 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.825596    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:04:04.857052 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.857026    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"]
Sep 17 19:04:04.864771 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.864749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afd974dd-e50b-4cbf-9125-3781732b732a-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.864908 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.864780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.864908 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.864801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afd974dd-e50b-4cbf-9125-3781732b732a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.864908 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.864843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkng\" (UniqueName: \"kubernetes.io/projected/afd974dd-e50b-4cbf-9125-3781732b732a-kube-api-access-smkng\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.864908 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.864881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.965489 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.965465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.965596 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.965494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afd974dd-e50b-4cbf-9125-3781732b732a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.965596 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.965518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkng\" (UniqueName: \"kubernetes.io/projected/afd974dd-e50b-4cbf-9125-3781732b732a-kube-api-access-smkng\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.965724 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.965646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.965724 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.965708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afd974dd-e50b-4cbf-9125-3781732b732a-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.965925 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.965902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afd974dd-e50b-4cbf-9125-3781732b732a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.966204 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.966183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.966274 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.966228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.968134 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.968116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afd974dd-e50b-4cbf-9125-3781732b732a-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:04.978330 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:04.978310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkng\" (UniqueName: \"kubernetes.io/projected/afd974dd-e50b-4cbf-9125-3781732b732a-kube-api-access-smkng\") pod \"isvc-canary-promote-predictor-5f4f848c44-wskmr\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:05.127598 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:05.127537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:05.199207 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:05.199182    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98" path="/var/lib/kubelet/pods/d77561b9-fc4e-41b5-bddd-0a2ba3ba8c98/volumes"
Sep 17 19:04:05.412426 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:05.412347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"]
Sep 17 19:04:05.415081 ip-10-0-136-86 kubenswrapper[2566]: W0917 19:04:05.415051    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd974dd_e50b_4cbf_9125_3781732b732a.slice/crio-0e3ad104f9d54d93b4b9a81f020752790ca2f6dab64e948c486d06ff084b4b8a WatchSource:0}: Error finding container 0e3ad104f9d54d93b4b9a81f020752790ca2f6dab64e948c486d06ff084b4b8a: Status 404 returned error can't find the container with id 0e3ad104f9d54d93b4b9a81f020752790ca2f6dab64e948c486d06ff084b4b8a
Sep 17 19:04:05.741816 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:05.741733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerStarted","Data":"57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2"}
Sep 17 19:04:05.741816 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:05.741789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerStarted","Data":"0e3ad104f9d54d93b4b9a81f020752790ca2f6dab64e948c486d06ff084b4b8a"}
Sep 17 19:04:09.759996 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:09.759962    2566 generic.go:356] "Generic (PLEG): container finished" podID="afd974dd-e50b-4cbf-9125-3781732b732a" containerID="57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2" exitCode=0
Sep 17 19:04:09.760303 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:09.760038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerDied","Data":"57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2"}
Sep 17 19:04:10.766506 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:10.766466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerStarted","Data":"2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471"}
Sep 17 19:04:10.766900 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:10.766537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerStarted","Data":"97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98"}
Sep 17 19:04:10.766900 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:10.766794    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:10.766981 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:10.766955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:10.767900 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:10.767877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:04:11.771005 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:11.770968    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:04:16.775364 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:16.775335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:04:16.775869 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:16.775814    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:04:16.805260 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:16.805217    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podStartSLOduration=8.744127132 podStartE2EDuration="12.805204976s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.061077844s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:10.792608269 +0000 UTC m=+1527.645312514" watchObservedRunningTime="2026-09-17 19:04:16.805204976 +0000 UTC m=+1533.657909219"
Sep 17 19:04:26.776572 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:26.776533    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:04:36.776246 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:36.776204    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:04:46.776559 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:46.776516    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:04:56.776260 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:04:56.776221    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:05:06.776306 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:06.776263    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:05:16.775783 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:16.775743    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:05:17.193958 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:17.193923    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:05:27.198175 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:27.198094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:05:28.045896 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:28.045865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"]
Sep 17 19:05:28.072184 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:28.070213    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container" containerID="cri-o://97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98" gracePeriod=30
Sep 17 19:05:28.072184 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:28.070469    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kube-rbac-proxy" containerID="cri-o://2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471" gracePeriod=30
Sep 17 19:05:29.076072 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:29.076038    2566 generic.go:356] "Generic (PLEG): container finished" podID="afd974dd-e50b-4cbf-9125-3781732b732a" containerID="2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471" exitCode=2
Sep 17 19:05:29.076439 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:29.076113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerDied","Data":"2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471"}
Sep 17 19:05:31.771928 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:31.771890    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 19:05:32.007256 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.007234    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:05:32.088466 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.088438    2566 generic.go:356] "Generic (PLEG): container finished" podID="afd974dd-e50b-4cbf-9125-3781732b732a" containerID="97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98" exitCode=0
Sep 17 19:05:32.088623 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.088498    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"
Sep 17 19:05:32.088623 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.088509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerDied","Data":"97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98"}
Sep 17 19:05:32.088623 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.088563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr" event={"ID":"afd974dd-e50b-4cbf-9125-3781732b732a","Type":"ContainerDied","Data":"0e3ad104f9d54d93b4b9a81f020752790ca2f6dab64e948c486d06ff084b4b8a"}
Sep 17 19:05:32.088623 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.088591    2566 scope.go:117] "RemoveContainer" containerID="2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471"
Sep 17 19:05:32.096291 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.096276    2566 scope.go:117] "RemoveContainer" containerID="97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98"
Sep 17 19:05:32.098325 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smkng\" (UniqueName: \"kubernetes.io/projected/afd974dd-e50b-4cbf-9125-3781732b732a-kube-api-access-smkng\") pod \"afd974dd-e50b-4cbf-9125-3781732b732a\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") "
Sep 17 19:05:32.098413 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afd974dd-e50b-4cbf-9125-3781732b732a-kserve-provision-location\") pod \"afd974dd-e50b-4cbf-9125-3781732b732a\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") "
Sep 17 19:05:32.098413 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098369    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afd974dd-e50b-4cbf-9125-3781732b732a-proxy-tls\") pod \"afd974dd-e50b-4cbf-9125-3781732b732a\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") "
Sep 17 19:05:32.098413 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-kube-rbac-proxy-sar-config\") pod \"afd974dd-e50b-4cbf-9125-3781732b732a\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") "
Sep 17 19:05:32.098596 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-cabundle-cert\") pod \"afd974dd-e50b-4cbf-9125-3781732b732a\" (UID: \"afd974dd-e50b-4cbf-9125-3781732b732a\") "
Sep 17 19:05:32.098754 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098667    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd974dd-e50b-4cbf-9125-3781732b732a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afd974dd-e50b-4cbf-9125-3781732b732a" (UID: "afd974dd-e50b-4cbf-9125-3781732b732a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:32.098869 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afd974dd-e50b-4cbf-9125-3781732b732a" (UID: "afd974dd-e50b-4cbf-9125-3781732b732a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:32.098869 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.098838    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afd974dd-e50b-4cbf-9125-3781732b732a" (UID: "afd974dd-e50b-4cbf-9125-3781732b732a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:32.100389 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.100370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd974dd-e50b-4cbf-9125-3781732b732a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afd974dd-e50b-4cbf-9125-3781732b732a" (UID: "afd974dd-e50b-4cbf-9125-3781732b732a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:32.100642 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.100622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd974dd-e50b-4cbf-9125-3781732b732a-kube-api-access-smkng" (OuterVolumeSpecName: "kube-api-access-smkng") pod "afd974dd-e50b-4cbf-9125-3781732b732a" (UID: "afd974dd-e50b-4cbf-9125-3781732b732a"). InnerVolumeSpecName "kube-api-access-smkng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:05:32.113764 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.113742    2566 scope.go:117] "RemoveContainer" containerID="57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2"
Sep 17 19:05:32.120927 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.120910    2566 scope.go:117] "RemoveContainer" containerID="2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471"
Sep 17 19:05:32.121185 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:05:32.121163    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471\": container with ID starting with 2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471 not found: ID does not exist" containerID="2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471"
Sep 17 19:05:32.121272 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.121197    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471"} err="failed to get container status \"2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471\": rpc error: code = NotFound desc = could not find container \"2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471\": container with ID starting with 2b47827050f08508fe25e2ec1c3150740a6ea0a5bfc5365bce5ab42443137471 not found: ID does not exist"
Sep 17 19:05:32.121272 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.121215    2566 scope.go:117] "RemoveContainer" containerID="97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98"
Sep 17 19:05:32.121442 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:05:32.121424    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98\": container with ID starting with 97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98 not found: ID does not exist" containerID="97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98"
Sep 17 19:05:32.121515 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.121451    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98"} err="failed to get container status \"97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98\": rpc error: code = NotFound desc = could not find container \"97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98\": container with ID starting with 97dbb5766bc4f6baef445d3b02793337451385c6bb3128536906d8c667d04c98 not found: ID does not exist"
Sep 17 19:05:32.121515 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.121472    2566 scope.go:117] "RemoveContainer" containerID="57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2"
Sep 17 19:05:32.121755 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:05:32.121737    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2\": container with ID starting with 57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2 not found: ID does not exist" containerID="57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2"
Sep 17 19:05:32.121793 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.121759    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2"} err="failed to get container status \"57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2\": rpc error: code = NotFound desc = could not find container \"57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2\": container with ID starting with 57ac7da62f0e11373a11e30b9c532a79ef940268cd29553ec578dd07912da1f2 not found: ID does not exist"
Sep 17 19:05:32.199131 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.199111    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afd974dd-e50b-4cbf-9125-3781732b732a-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:05:32.199131 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.199129    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afd974dd-e50b-4cbf-9125-3781732b732a-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:05:32.199261 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.199140    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:05:32.199261 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.199150    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afd974dd-e50b-4cbf-9125-3781732b732a-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:05:32.199261 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.199158    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smkng\" (UniqueName: \"kubernetes.io/projected/afd974dd-e50b-4cbf-9125-3781732b732a-kube-api-access-smkng\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:05:32.410913 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.410887    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"]
Sep 17 19:05:32.416800 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:32.416780    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-wskmr"]
Sep 17 19:05:33.204252 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:33.204218    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" path="/var/lib/kubelet/pods/afd974dd-e50b-4cbf-9125-3781732b732a/volumes"
Sep 17 19:05:34.669872 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.669815    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"]
Sep 17 19:05:34.670241 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670159    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="storage-initializer"
Sep 17 19:05:34.670241 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670171    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="storage-initializer"
Sep 17 19:05:34.670241 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670179    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container"
Sep 17 19:05:34.670241 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670185    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container"
Sep 17 19:05:34.670241 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670215    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kube-rbac-proxy"
Sep 17 19:05:34.670241 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670221    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kube-rbac-proxy"
Sep 17 19:05:34.670428 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kube-rbac-proxy"
Sep 17 19:05:34.670428 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.670282    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd974dd-e50b-4cbf-9125-3781732b732a" containerName="kserve-container"
Sep 17 19:05:34.675119 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.675096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.676284 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.676258    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"]
Sep 17 19:05:34.678345 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:05:34.678451 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:05:34.678451 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:05:34.678576 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678480    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z5lc7\""
Sep 17 19:05:34.678576 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678535    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:05:34.678576 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678552    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 19:05:34.678724 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.678619    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:05:34.715403 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.715377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a08d45-8c25-4076-8df9-0453fd146351-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.715538 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.715419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a08d45-8c25-4076-8df9-0453fd146351-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.715538 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.715444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4tc\" (UniqueName: \"kubernetes.io/projected/21a08d45-8c25-4076-8df9-0453fd146351-kube-api-access-2h4tc\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.715538 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.715487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.715538 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.715506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.816500 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.816474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4tc\" (UniqueName: \"kubernetes.io/projected/21a08d45-8c25-4076-8df9-0453fd146351-kube-api-access-2h4tc\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.816599 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.816509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.816599 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.816529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.816599 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.816560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a08d45-8c25-4076-8df9-0453fd146351-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.816599 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.816589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a08d45-8c25-4076-8df9-0453fd146351-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.817004 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.816984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a08d45-8c25-4076-8df9-0453fd146351-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.817236 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.817221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.817312 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.817291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.819069 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.819052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a08d45-8c25-4076-8df9-0453fd146351-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.826625 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.826600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4tc\" (UniqueName: \"kubernetes.io/projected/21a08d45-8c25-4076-8df9-0453fd146351-kube-api-access-2h4tc\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:34.985966 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:34.985915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:35.259370 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:35.259346    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"]
Sep 17 19:05:35.262049 ip-10-0-136-86 kubenswrapper[2566]: W0917 19:05:35.262018    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a08d45_8c25_4076_8df9_0453fd146351.slice/crio-7a84435bfa1f4b7157d97340e44a3833f7274f7c32d83a938e590f44b3214364 WatchSource:0}: Error finding container 7a84435bfa1f4b7157d97340e44a3833f7274f7c32d83a938e590f44b3214364: Status 404 returned error can't find the container with id 7a84435bfa1f4b7157d97340e44a3833f7274f7c32d83a938e590f44b3214364
Sep 17 19:05:36.108213 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:36.108177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerStarted","Data":"d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50"}
Sep 17 19:05:36.108586 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:36.108244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerStarted","Data":"7a84435bfa1f4b7157d97340e44a3833f7274f7c32d83a938e590f44b3214364"}
Sep 17 19:05:40.125690 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:40.125653    2566 generic.go:356] "Generic (PLEG): container finished" podID="21a08d45-8c25-4076-8df9-0453fd146351" containerID="d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50" exitCode=0
Sep 17 19:05:40.126096 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:40.125725    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerDied","Data":"d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50"}
Sep 17 19:05:41.131699 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:41.131661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerStarted","Data":"b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687"}
Sep 17 19:05:41.132086 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:41.131754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerStarted","Data":"0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a"}
Sep 17 19:05:41.132086 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:41.132001    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:41.159683 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:41.159632    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podStartSLOduration=3.24644967 podStartE2EDuration="7.159616669s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913166999s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:41.157510387 +0000 UTC m=+1618.010214631" watchObservedRunningTime="2026-09-17 19:05:41.159616669 +0000 UTC m=+1618.012320965"
Sep 17 19:05:42.136222 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:42.136189    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:42.137404 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:42.137375    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:05:43.140716 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:43.140674    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:05:48.144847 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:48.144807    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:05:48.145277 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:48.145247    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:05:58.145375 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:05:58.145330    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:06:08.145846 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:06:08.145801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:06:18.145713 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:06:18.145664    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:06:28.145748 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:06:28.145708    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:06:38.145935 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:06:38.145898    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:06:48.145629 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:06:48.145589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:06:58.146866 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:06:58.146757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:07:04.851175 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:04.851144    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"]
Sep 17 19:07:04.851667 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:04.851418    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" containerID="cri-o://0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a" gracePeriod=30
Sep 17 19:07:04.851667 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:04.851448    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kube-rbac-proxy" containerID="cri-o://b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687" gracePeriod=30
Sep 17 19:07:05.471394 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:05.471361    2566 generic.go:356] "Generic (PLEG): container finished" podID="21a08d45-8c25-4076-8df9-0453fd146351" containerID="b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687" exitCode=2
Sep 17 19:07:05.471565 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:05.471402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerDied","Data":"b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687"}
Sep 17 19:07:08.141604 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.141562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 19:07:08.146037 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.146007    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:07:08.794281 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.794259    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:07:08.903576 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a08d45-8c25-4076-8df9-0453fd146351-proxy-tls\") pod \"21a08d45-8c25-4076-8df9-0453fd146351\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") "
Sep 17 19:07:08.903576 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903555    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a08d45-8c25-4076-8df9-0453fd146351-kserve-provision-location\") pod \"21a08d45-8c25-4076-8df9-0453fd146351\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") "
Sep 17 19:07:08.903576 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-kube-rbac-proxy-sar-config\") pod \"21a08d45-8c25-4076-8df9-0453fd146351\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") "
Sep 17 19:07:08.903796 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h4tc\" (UniqueName: \"kubernetes.io/projected/21a08d45-8c25-4076-8df9-0453fd146351-kube-api-access-2h4tc\") pod \"21a08d45-8c25-4076-8df9-0453fd146351\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") "
Sep 17 19:07:08.903796 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-cabundle-cert\") pod \"21a08d45-8c25-4076-8df9-0453fd146351\" (UID: \"21a08d45-8c25-4076-8df9-0453fd146351\") "
Sep 17 19:07:08.903935 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a08d45-8c25-4076-8df9-0453fd146351-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21a08d45-8c25-4076-8df9-0453fd146351" (UID: "21a08d45-8c25-4076-8df9-0453fd146351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:08.903935 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.903896    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21a08d45-8c25-4076-8df9-0453fd146351" (UID: "21a08d45-8c25-4076-8df9-0453fd146351"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:08.904065 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.904045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21a08d45-8c25-4076-8df9-0453fd146351" (UID: "21a08d45-8c25-4076-8df9-0453fd146351"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:08.905584 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.905557    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a08d45-8c25-4076-8df9-0453fd146351-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21a08d45-8c25-4076-8df9-0453fd146351" (UID: "21a08d45-8c25-4076-8df9-0453fd146351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:08.905678 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:08.905612    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a08d45-8c25-4076-8df9-0453fd146351-kube-api-access-2h4tc" (OuterVolumeSpecName: "kube-api-access-2h4tc") pod "21a08d45-8c25-4076-8df9-0453fd146351" (UID: "21a08d45-8c25-4076-8df9-0453fd146351"). InnerVolumeSpecName "kube-api-access-2h4tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:09.004638 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.004614    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a08d45-8c25-4076-8df9-0453fd146351-proxy-tls\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.004638 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.004635    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a08d45-8c25-4076-8df9-0453fd146351-kserve-provision-location\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.004859 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.004644    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.004859 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.004653    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4tc\" (UniqueName: \"kubernetes.io/projected/21a08d45-8c25-4076-8df9-0453fd146351-kube-api-access-2h4tc\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.004859 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.004662    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a08d45-8c25-4076-8df9-0453fd146351-cabundle-cert\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:07:09.489396 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.489326    2566 generic.go:356] "Generic (PLEG): container finished" podID="21a08d45-8c25-4076-8df9-0453fd146351" containerID="0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a" exitCode=0
Sep 17 19:07:09.489396 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.489380    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"
Sep 17 19:07:09.489396 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.489383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerDied","Data":"0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a"}
Sep 17 19:07:09.489944 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.489439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv" event={"ID":"21a08d45-8c25-4076-8df9-0453fd146351","Type":"ContainerDied","Data":"7a84435bfa1f4b7157d97340e44a3833f7274f7c32d83a938e590f44b3214364"}
Sep 17 19:07:09.489944 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.489467    2566 scope.go:117] "RemoveContainer" containerID="b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687"
Sep 17 19:07:09.497440 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.497420    2566 scope.go:117] "RemoveContainer" containerID="0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a"
Sep 17 19:07:09.504634 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.504617    2566 scope.go:117] "RemoveContainer" containerID="d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50"
Sep 17 19:07:09.511586 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.511568    2566 scope.go:117] "RemoveContainer" containerID="b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687"
Sep 17 19:07:09.511885 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:07:09.511867    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687\": container with ID starting with b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687 not found: ID does not exist" containerID="b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687"
Sep 17 19:07:09.511942 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.511892    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687"} err="failed to get container status \"b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687\": rpc error: code = NotFound desc = could not find container \"b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687\": container with ID starting with b9f706d5b41bb40f2ea9a2e13b16cd70de352bc32e140a598b579f9bc76e0687 not found: ID does not exist"
Sep 17 19:07:09.511942 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.511909    2566 scope.go:117] "RemoveContainer" containerID="0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a"
Sep 17 19:07:09.512187 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:07:09.512167    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a\": container with ID starting with 0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a not found: ID does not exist" containerID="0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a"
Sep 17 19:07:09.512240 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.512193    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a"} err="failed to get container status \"0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a\": rpc error: code = NotFound desc = could not find container \"0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a\": container with ID starting with 0dd22dba4c569638618240cc606a6709c5830a6d10090c2c9e49f1519b67664a not found: ID does not exist"
Sep 17 19:07:09.512240 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.512209    2566 scope.go:117] "RemoveContainer" containerID="d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50"
Sep 17 19:07:09.512441 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:07:09.512424    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50\": container with ID starting with d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50 not found: ID does not exist" containerID="d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50"
Sep 17 19:07:09.512482 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:09.512444    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50"} err="failed to get container status \"d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50\": rpc error: code = NotFound desc = could not find container \"d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50\": container with ID starting with d18ef10db9d35a6220c0073982af440388445e702b6f7ec458c494d989314d50 not found: ID does not exist"
Sep 17 19:07:11.348555 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:11.348521    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"]
Sep 17 19:07:11.909766 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:11.909736    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-8mtsv"]
Sep 17 19:07:13.198350 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:07:13.198316    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a08d45-8c25-4076-8df9-0453fd146351" path="/var/lib/kubelet/pods/21a08d45-8c25-4076-8df9-0453fd146351/volumes"
Sep 17 19:08:45.260933 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:08:45.260907    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 19:08:45.263717 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:08:45.263689    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 19:09:16.585148 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585107    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7srn5/must-gather-d5hd5"]
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585424    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585437    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585453    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kube-rbac-proxy"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585459    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kube-rbac-proxy"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585469    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="storage-initializer"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585475    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="storage-initializer"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585520    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kserve-container"
Sep 17 19:09:16.585663 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.585529    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a08d45-8c25-4076-8df9-0453fd146351" containerName="kube-rbac-proxy"
Sep 17 19:09:16.588517 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.588501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.594068 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.594034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7srn5\"/\"kube-root-ca.crt\""
Sep 17 19:09:16.594790 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.594772    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7srn5\"/\"default-dockercfg-lnxq4\""
Sep 17 19:09:16.594790 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.594781    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7srn5\"/\"openshift-service-ca.crt\""
Sep 17 19:09:16.603344 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.603314    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7srn5/must-gather-d5hd5"]
Sep 17 19:09:16.712579 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.712552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrcsz\" (UniqueName: \"kubernetes.io/projected/bbdea042-6a51-4730-b5ce-6be1916838b2-kube-api-access-vrcsz\") pod \"must-gather-d5hd5\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") " pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.712719 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.712586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbdea042-6a51-4730-b5ce-6be1916838b2-must-gather-output\") pod \"must-gather-d5hd5\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") " pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.813897 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.813857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrcsz\" (UniqueName: \"kubernetes.io/projected/bbdea042-6a51-4730-b5ce-6be1916838b2-kube-api-access-vrcsz\") pod \"must-gather-d5hd5\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") " pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.814036 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.813908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbdea042-6a51-4730-b5ce-6be1916838b2-must-gather-output\") pod \"must-gather-d5hd5\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") " pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.814236 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.814218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbdea042-6a51-4730-b5ce-6be1916838b2-must-gather-output\") pod \"must-gather-d5hd5\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") " pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.822777 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.822748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrcsz\" (UniqueName: \"kubernetes.io/projected/bbdea042-6a51-4730-b5ce-6be1916838b2-kube-api-access-vrcsz\") pod \"must-gather-d5hd5\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") " pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:16.913380 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:16.913311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:17.188582 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:17.188544    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7srn5/must-gather-d5hd5"]
Sep 17 19:09:17.190202 ip-10-0-136-86 kubenswrapper[2566]: W0917 19:09:17.190167    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbdea042_6a51_4730_b5ce_6be1916838b2.slice/crio-d85d5eb1a33937acc3e97a98938b2f7cd7436000e8bbfd73f0dbcccf17c85365 WatchSource:0}: Error finding container d85d5eb1a33937acc3e97a98938b2f7cd7436000e8bbfd73f0dbcccf17c85365: Status 404 returned error can't find the container with id d85d5eb1a33937acc3e97a98938b2f7cd7436000e8bbfd73f0dbcccf17c85365
Sep 17 19:09:17.191911 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:17.191894    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:09:18.001871 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:18.001813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7srn5/must-gather-d5hd5" event={"ID":"bbdea042-6a51-4730-b5ce-6be1916838b2","Type":"ContainerStarted","Data":"d85d5eb1a33937acc3e97a98938b2f7cd7436000e8bbfd73f0dbcccf17c85365"}
Sep 17 19:09:24.034016 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:24.033925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7srn5/must-gather-d5hd5" event={"ID":"bbdea042-6a51-4730-b5ce-6be1916838b2","Type":"ContainerStarted","Data":"19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca"}
Sep 17 19:09:24.034016 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:24.033968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7srn5/must-gather-d5hd5" event={"ID":"bbdea042-6a51-4730-b5ce-6be1916838b2","Type":"ContainerStarted","Data":"67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"}
Sep 17 19:09:24.051808 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:24.051744    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7srn5/must-gather-d5hd5" podStartSLOduration=1.921036298 podStartE2EDuration="8.051724222s" totalImagesPullingTime="6.130687924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:24.050239008 +0000 UTC m=+1840.902943254" watchObservedRunningTime="2026-09-17 19:09:24.051724222 +0000 UTC m=+1840.904428466"
Sep 17 19:09:47.137078 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:47.137037    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerID="67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7" exitCode=0
Sep 17 19:09:47.137552 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:47.137108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7srn5/must-gather-d5hd5" event={"ID":"bbdea042-6a51-4730-b5ce-6be1916838b2","Type":"ContainerDied","Data":"67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"}
Sep 17 19:09:47.137552 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:47.137414    2566 scope.go:117] "RemoveContainer" containerID="67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"
Sep 17 19:09:47.432627 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:47.432543    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7srn5_must-gather-d5hd5_bbdea042-6a51-4730-b5ce-6be1916838b2/gather/0.log"
Sep 17 19:09:50.838778 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:50.838747    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bvtmd_853ece52-4596-4c55-83af-da02fc283b14/global-pull-secret-syncer/0.log"
Sep 17 19:09:50.994529 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:50.994503    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lkmvt_2e8be8b1-3cb1-4773-ac22-e32387848821/konnectivity-agent/0.log"
Sep 17 19:09:51.115442 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:51.115366    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-86.ec2.internal_feb5980f217da1fed6de338fd44f47c5/haproxy/0.log"
Sep 17 19:09:52.910229 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:52.910190    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7srn5/must-gather-d5hd5"]
Sep 17 19:09:52.910670 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:52.910378    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-7srn5/must-gather-d5hd5" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="copy" containerID="cri-o://19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca" gracePeriod=2
Sep 17 19:09:52.914786 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:52.914464    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7srn5/must-gather-d5hd5"]
Sep 17 19:09:53.133512 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.133493    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7srn5_must-gather-d5hd5_bbdea042-6a51-4730-b5ce-6be1916838b2/copy/0.log"
Sep 17 19:09:53.135904 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.135889    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:53.138640 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:09:53.138620    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d5hd5\" is forbidden: User \"system:node:ip-10-0-136-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7srn5\": no relationship found between node 'ip-10-0-136-86.ec2.internal' and this object" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:53.165551 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.165500    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7srn5_must-gather-d5hd5_bbdea042-6a51-4730-b5ce-6be1916838b2/copy/0.log"
Sep 17 19:09:53.165847 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.165811    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerID="19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca" exitCode=143
Sep 17 19:09:53.165928 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.165906    2566 scope.go:117] "RemoveContainer" containerID="19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca"
Sep 17 19:09:53.165986 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.165863    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:53.168368 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:09:53.168347    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d5hd5\" is forbidden: User \"system:node:ip-10-0-136-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7srn5\": no relationship found between node 'ip-10-0-136-86.ec2.internal' and this object" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" pod="openshift-must-gather-7srn5/must-gather-d5hd5"
Sep 17 19:09:53.173045 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.173028    2566 scope.go:117] "RemoveContainer" containerID="67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"
Sep 17 19:09:53.185111 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.185096    2566 scope.go:117] "RemoveContainer" containerID="19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca"
Sep 17 19:09:53.185357 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:09:53.185342    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca\": container with ID starting with 19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca not found: ID does not exist" containerID="19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca"
Sep 17 19:09:53.185402 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.185366    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca"} err="failed to get container status \"19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca\": rpc error: code = NotFound desc = could not find container \"19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca\": container with ID starting with 19f4d6fafaba56888edc3cad00ff74051f3c88436ba39238b2ee6e128c6888ca not found: ID does not exist"
Sep 17 19:09:53.185402 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.185381    2566 scope.go:117] "RemoveContainer" containerID="67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"
Sep 17 19:09:53.185592 ip-10-0-136-86 kubenswrapper[2566]: E0917 19:09:53.185577    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7\": container with ID starting with 67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7 not found: ID does not exist" containerID="67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"
Sep 17 19:09:53.185639 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.185596    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7"} err="failed to get container status \"67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7\": rpc error: code = NotFound desc = could not find container \"67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7\": container with ID starting with 67ed60cc8952f209ae1b63050a8ecd71e6c78962ab8192494f6b9be8a68c35e7 not found: ID does not exist"
Sep 17 19:09:53.223526 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.223504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrcsz\" (UniqueName: \"kubernetes.io/projected/bbdea042-6a51-4730-b5ce-6be1916838b2-kube-api-access-vrcsz\") pod \"bbdea042-6a51-4730-b5ce-6be1916838b2\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") "
Sep 17 19:09:53.223608 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.223539    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbdea042-6a51-4730-b5ce-6be1916838b2-must-gather-output\") pod \"bbdea042-6a51-4730-b5ce-6be1916838b2\" (UID: \"bbdea042-6a51-4730-b5ce-6be1916838b2\") "
Sep 17 19:09:53.225524 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.225496    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdea042-6a51-4730-b5ce-6be1916838b2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bbdea042-6a51-4730-b5ce-6be1916838b2" (UID: "bbdea042-6a51-4730-b5ce-6be1916838b2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:53.225600 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.225505    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdea042-6a51-4730-b5ce-6be1916838b2-kube-api-access-vrcsz" (OuterVolumeSpecName: "kube-api-access-vrcsz") pod "bbdea042-6a51-4730-b5ce-6be1916838b2" (UID: "bbdea042-6a51-4730-b5ce-6be1916838b2"). InnerVolumeSpecName "kube-api-access-vrcsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:53.324847 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.324811    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrcsz\" (UniqueName: \"kubernetes.io/projected/bbdea042-6a51-4730-b5ce-6be1916838b2-kube-api-access-vrcsz\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:09:53.324930 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:53.324850    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbdea042-6a51-4730-b5ce-6be1916838b2-must-gather-output\") on node \"ip-10-0-136-86.ec2.internal\" DevicePath \"\""
Sep 17 19:09:55.202718 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.202688    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" path="/var/lib/kubelet/pods/bbdea042-6a51-4730-b5ce-6be1916838b2/volumes"
Sep 17 19:09:55.296599 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.296569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-z5jrk_3a7cdc2b-fe9c-483b-917b-bc76138e839e/kube-state-metrics/0.log"
Sep 17 19:09:55.317759 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.317725    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-z5jrk_3a7cdc2b-fe9c-483b-917b-bc76138e839e/kube-rbac-proxy-main/0.log"
Sep 17 19:09:55.344783 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.344763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-z5jrk_3a7cdc2b-fe9c-483b-917b-bc76138e839e/kube-rbac-proxy-self/0.log"
Sep 17 19:09:55.379227 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.379212    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bb5cf5c69-nx7wl_9f99c367-a9cb-4657-8495-7b144b21737b/metrics-server/0.log"
Sep 17 19:09:55.410776 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.410759    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-6wdp7_fdfbf636-a10e-4384-b47c-efdadeafdb88/monitoring-plugin/0.log"
Sep 17 19:09:55.603742 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.603718    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7w59_9d6c1b83-d256-498a-a829-91326947139c/node-exporter/0.log"
Sep 17 19:09:55.623210 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.623189    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7w59_9d6c1b83-d256-498a-a829-91326947139c/kube-rbac-proxy/0.log"
Sep 17 19:09:55.643166 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.643150    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7w59_9d6c1b83-d256-498a-a829-91326947139c/init-textfile/0.log"
Sep 17 19:09:55.785008 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.784989    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/prometheus/0.log"
Sep 17 19:09:55.805380 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.805357    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/config-reloader/0.log"
Sep 17 19:09:55.828246 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.828224    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/thanos-sidecar/0.log"
Sep 17 19:09:55.851697 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.851673    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/kube-rbac-proxy-web/0.log"
Sep 17 19:09:55.879811 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.879749    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/kube-rbac-proxy/0.log"
Sep 17 19:09:55.903769 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.903749    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/kube-rbac-proxy-thanos/0.log"
Sep 17 19:09:55.926198 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.926181    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9bdc87-afdb-4b90-8bf3-7281ec5e9d37/init-config-reloader/0.log"
Sep 17 19:09:55.965883 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.965866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-lp8m8_4adea632-e3b6-44f6-9bcc-ed0ce9862378/prometheus-operator/0.log"
Sep 17 19:09:55.989483 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:55.989466    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-lp8m8_4adea632-e3b6-44f6-9bcc-ed0ce9862378/kube-rbac-proxy/0.log"
Sep 17 19:09:57.614033 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:57.613999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-jcfxc_3d8cb0ee-0042-48b9-91f4-3f29fee8230e/networking-console-plugin/0.log"
Sep 17 19:09:58.542702 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.542661    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-ws6vr_36d6cf8b-6e42-4dc2-bd08-dce0b6159ff5/download-server/0.log"
Sep 17 19:09:58.962627 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.962588    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"]
Sep 17 19:09:58.963110 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.963061    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="gather"
Sep 17 19:09:58.963110 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.963083    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="gather"
Sep 17 19:09:58.963110 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.963094    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="copy"
Sep 17 19:09:58.963110 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.963102    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="copy"
Sep 17 19:09:58.963319 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.963200    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="gather"
Sep 17 19:09:58.963319 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.963214    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbdea042-6a51-4730-b5ce-6be1916838b2" containerName="copy"
Sep 17 19:09:58.966314 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.966292    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:58.969245 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.969222    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-59s4v\"/\"default-dockercfg-v4fdh\""
Sep 17 19:09:58.969713 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.969696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59s4v\"/\"kube-root-ca.crt\""
Sep 17 19:09:58.970251 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.970230    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59s4v\"/\"openshift-service-ca.crt\""
Sep 17 19:09:58.975798 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:58.975775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"]
Sep 17 19:09:59.068059 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.068027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-lib-modules\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.068196 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.068072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-proc\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.068196 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.068104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tsfs\" (UniqueName: \"kubernetes.io/projected/26756dce-a1fa-4b40-b160-b5c24d5c4a67-kube-api-access-7tsfs\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.068196 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.068121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-podres\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.068196 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.068185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-sys\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.168943 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.168916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-lib-modules\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169075 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.168951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-proc\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169075 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.168994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tsfs\" (UniqueName: \"kubernetes.io/projected/26756dce-a1fa-4b40-b160-b5c24d5c4a67-kube-api-access-7tsfs\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169075 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.169020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-podres\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169075 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.169057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-sys\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169247 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.169067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-proc\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169247 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.169109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-lib-modules\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169247 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.169149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-sys\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.169247 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.169185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26756dce-a1fa-4b40-b160-b5c24d5c4a67-podres\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.179210 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.179184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tsfs\" (UniqueName: \"kubernetes.io/projected/26756dce-a1fa-4b40-b160-b5c24d5c4a67-kube-api-access-7tsfs\") pod \"perf-node-gather-daemonset-rfzcc\" (UID: \"26756dce-a1fa-4b40-b160-b5c24d5c4a67\") " pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.277740 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.277685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:09:59.548920 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:09:59.548896    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"]
Sep 17 19:09:59.551092 ip-10-0-136-86 kubenswrapper[2566]: W0917 19:09:59.551063    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26756dce_a1fa_4b40_b160_b5c24d5c4a67.slice/crio-3d3661585af175c726508b0057fe1ef2b5c8fd5603808cf1f1984f6ac11c7305 WatchSource:0}: Error finding container 3d3661585af175c726508b0057fe1ef2b5c8fd5603808cf1f1984f6ac11c7305: Status 404 returned error can't find the container with id 3d3661585af175c726508b0057fe1ef2b5c8fd5603808cf1f1984f6ac11c7305
Sep 17 19:10:00.049488 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.049446    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-khlll_d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb/dns/0.log"
Sep 17 19:10:00.071625 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.071599    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-khlll_d98d1e82-ee9a-4887-b9d3-9cb3a054b3cb/kube-rbac-proxy/0.log"
Sep 17 19:10:00.191912 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.191885    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q9dbd_79480ed8-4de0-47ef-80ba-80f25ce854f1/dns-node-resolver/0.log"
Sep 17 19:10:00.195048 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.195025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc" event={"ID":"26756dce-a1fa-4b40-b160-b5c24d5c4a67","Type":"ContainerStarted","Data":"d3a5ee2827f7ff6ec68130b27644851407e24beddeef0e5064d31e447cd86c2d"}
Sep 17 19:10:00.195189 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.195104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:10:00.195189 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.195123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc" event={"ID":"26756dce-a1fa-4b40-b160-b5c24d5c4a67","Type":"ContainerStarted","Data":"3d3661585af175c726508b0057fe1ef2b5c8fd5603808cf1f1984f6ac11c7305"}
Sep 17 19:10:00.212771 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.212733    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc" podStartSLOduration=2.212719705 podStartE2EDuration="2.212719705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:00.210959589 +0000 UTC m=+1877.063663846" watchObservedRunningTime="2026-09-17 19:10:00.212719705 +0000 UTC m=+1877.065423948"
Sep 17 19:10:00.617621 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.617591    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c74b8f954-cgwhn_107228bb-9cf8-425e-a692-f0fa85a8768e/registry/0.log"
Sep 17 19:10:00.657919 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:00.657897    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qr8gm_974224b4-09ad-4737-8b71-3df6f66f0dfd/node-ca/0.log"
Sep 17 19:10:01.654841 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:01.654797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b78d2_83139208-88be-4b5e-84ea-365d834265ec/serve-healthcheck-canary/0.log"
Sep 17 19:10:02.060722 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:02.060676    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nfkgq_198c3a10-a9d4-40a5-b36a-d644cb3b548a/kube-rbac-proxy/0.log"
Sep 17 19:10:02.079649 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:02.079625    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nfkgq_198c3a10-a9d4-40a5-b36a-d644cb3b548a/exporter/0.log"
Sep 17 19:10:02.101006 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:02.100984    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nfkgq_198c3a10-a9d4-40a5-b36a-d644cb3b548a/extractor/0.log"
Sep 17 19:10:04.954604 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:04.954576    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-s55th_5876b4df-6114-4f83-88a9-5d6afbcb440f/s3-init/0.log"
Sep 17 19:10:04.969069 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:04.969043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-s55th_5876b4df-6114-4f83-88a9-5d6afbcb440f/download-hf-model/0.log"
Sep 17 19:10:06.208401 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:06.208375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-59s4v/perf-node-gather-daemonset-rfzcc"
Sep 17 19:10:09.226037 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:09.225995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qs6vs_84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5/migrator/0.log"
Sep 17 19:10:09.246856 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:09.246816    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qs6vs_84a16572-8eb4-4eba-8a1a-70a6f7b9b5c5/graceful-termination/0.log"
Sep 17 19:10:09.535552 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:09.535469    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jfxvf_d81bfd12-ec20-4803-b867-dc1401885094/kube-storage-version-migrator-operator/1.log"
Sep 17 19:10:09.537195 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:09.537176    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jfxvf_d81bfd12-ec20-4803-b867-dc1401885094/kube-storage-version-migrator-operator/0.log"
Sep 17 19:10:10.839060 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.839036    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:10:10.863011 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.862987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/egress-router-binary-copy/0.log"
Sep 17 19:10:10.887325 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.887309    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/cni-plugins/0.log"
Sep 17 19:10:10.911629 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.911608    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/bond-cni-plugin/0.log"
Sep 17 19:10:10.936577 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.936553    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/routeoverride-cni/0.log"
Sep 17 19:10:10.963694 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.963678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/whereabouts-cni-bincopy/0.log"
Sep 17 19:10:10.984086 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:10.984067    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wj95n_e46de969-dafc-495c-a1df-6c72a069ff78/whereabouts-cni/0.log"
Sep 17 19:10:11.120989 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:11.120944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qtzwq_fc3335f9-1553-4f57-b95a-3cd8ac5069e3/kube-multus/0.log"
Sep 17 19:10:11.240640 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:11.240617    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9gtzs_1e4b8718-ff20-4f8b-bfee-853b19603fdc/network-metrics-daemon/0.log"
Sep 17 19:10:11.267166 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:11.267146    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9gtzs_1e4b8718-ff20-4f8b-bfee-853b19603fdc/kube-rbac-proxy/0.log"
Sep 17 19:10:12.506711 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.506685    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-controller/0.log"
Sep 17 19:10:12.523288 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.523253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/0.log"
Sep 17 19:10:12.523430 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.523294    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovn-acl-logging/1.log"
Sep 17 19:10:12.545770 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.545745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/kube-rbac-proxy-node/0.log"
Sep 17 19:10:12.568739 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.568719    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:10:12.586815 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.586789    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/northd/0.log"
Sep 17 19:10:12.606876 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.606857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/nbdb/0.log"
Sep 17 19:10:12.628036 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.628019    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/sbdb/0.log"
Sep 17 19:10:12.681384 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:12.681336    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xwfls_b0c1dd66-c309-436d-838a-f3b29ea4af63/ovnkube-controller/0.log"
Sep 17 19:10:13.672649 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:13.672618    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qfzqm_4e1e9522-f9b9-4675-a969-084168f1bb41/network-check-target-container/0.log"
Sep 17 19:10:14.535071 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:14.535042    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zjhmb_6249ccd6-6c9d-43d6-ae9d-61beefa0e564/iptables-alerter/0.log"
Sep 17 19:10:15.227846 ip-10-0-136-86 kubenswrapper[2566]: I0917 19:10:15.227802    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xb28h_6c8dd36c-edaa-439a-a035-815b938708bd/tuned/0.log"